Well guys, this is it, the last episode of Out of Frame. When I first joined the Foundation for Economic Education in 2016, my goal as Director of Media was to build up a creative team and an effective content strategy from scratch. At the time, Fee's YouTube channel had about 13,000 subscribers with barely any views and there hadn't been a new upload in a long time. So for the first year or so, I just started experimenting with a ton of different stuff. We did news briefs, animated shorts, a vlog series, a few different podcasts, a bunch of documentaries and even an educational series featuring my buddy Kevin Lieber from Vsauce 2 and Handmade Puppets. Hey, Kevin here. Some of that stuff worked pretty well and a couple of my documentaries won some awards but out of all that experimentation, there were really two shows that stood out. Common Sense Soapbox, which I created with Seamus Coughlin before Freedom Tunes really blew up and of course, Out of Frame, which started in 2017 with an eight minute episode about Star Wars. Everything I've done at Fee has always been a reflection of my creative vision but Out of Frame is a lot more directly me than anything else. To some extent, it's just the extension of the way I think about stories combined with my long-term interest in economics and political philosophy. Sure, the video essay format is uniquely suited to this kind of project but to be honest, most of what I do here isn't all that different from the conversations I have with friends whenever we hang out and watch movies. I think that's partly why this series works. The other reason it works is because since the beginning, my colleagues at Fee have trusted me to make content the way I want to make it. As much as some people in the comments have imagined that there's some big committee of overlords deciding what I say, the fact is there's never been much of a filter between me and you. That's true both in terms of how I choose topics, write episodes and edit the show and in terms of how I interact with people in the comments. Most nonprofits and for-profit companies I've worked for or with over the years are too afraid to allow creators that kind of freedom but it makes a big difference just like it does in every other area of life. And over the last five years, I've learned a lot about YouTube, about what you all care about and about the overall landscape of ideas that are taking hold in society. So that's what I want to talk about today on this very special final episode of Out of Frame. At the risk of this becoming too maudlin, I've been pretty amazed by all the comments I've gotten since I announced that I was leaving. Over the run of the show, I've tried my best to stay engaged and respond to as many people as I can. That's always meant trying to be grateful for all of you who have said kind things about episodes and also responding to criticism. Of course, with hundreds of thousands of comments since 2017, there's been quite a lot of both and there's really no way I could ever have replied to everything. That said, since my goal without a frame was and is to spark important conversations and get people thinking about ideas that are often unfamiliar and new, it's super important that we all keep talking to each other with at least a modicum of respect in the comments. I know that at least in a few cases, these conversations have made a significant difference. So that's one thing that I hope everyone will think about even after this series is gone. Talking to other people matters. Taking the time to actually communicate your ideas in a way that everyone can clearly understand and empathize with is critical. And that's going to be one of the most important skills any of us can develop right now because we're entering what I think is going to be a very difficult time. But it's also important to keep our own challenges in perspective. To that end, there's a movie I've never talked about on this series that I think is fitting for the final episode, Steven Spielberg's 1997 Academy Award-nominated film, Amistad. Based on horrifying true events, Amistad is the story of a group of illegally enslaved West Africans who managed to free themselves and kill most of their captors while being taken from Cuba to the United States on the Spanish ship La Amistad in 1839. After the Mendee tribesmen led by Sang B. Pia, later known as Joseph Sincay, get control of the ship, they demand to be taken home to what is now Sierra Leone. But the surviving crewmen take them north instead, where they're discovered off the coast of Connecticut by the Navy brig ship, the USS Washington. This sets off a complex legal case that becomes the central plot of Spielberg's film. Since none of the captive Africans spoke English and no one in Connecticut could translate for them, the whole situation turned into an epic international conflict. The slavers, backed by the Queen of Spain, claimed that the Africans were their property and needed to be returned to Cuba where they supposedly came from. Some of the naval officers who captured the Amistad claimed the ship and the men and women on it as their property by right of salvage. And of course, the Mendee tribesmen insist that they are rightfully free people, which by international law could very well be true. You see, at the time, the British empire had already outlawed slavery and engaged its Navy to crack down on the transatlantic slave trade in their territories. As a result, no African from a British colony was legally allowed to be captured and transported across the Atlantic Ocean. Yet the abduction of freemen from the British protector of Sierra Leone and their illegal transportation to the new world, as described by Sincay, is not unheard of, is it? Not even unusual, regrettably. Unfortunately, it was still illegal to transport people who were enslaved outside of the British empire, including Cuba. So this rule created some truly grotesque incentives, which were brought to light in the Amistad case. For one thing, slavers began keeping two different sets of records so that they could credibly lie about the origin of the human beings imprisoned on their ships. In the case of the Amistad, the Mendee slaves had been illegally captured in Sierra Leone and were sold at the Lomboco slave fortress to the Portuguese ship to Cora, which recorded them as having come from Cuba. But worse, if slavers felt like they were at risk of being boarded by the British Navy, they would sometimes chain their captives together and push them overboard, drowning dozens of people just to hide the evidence of their crimes. That also happens in Amistad. Ultimately, the court case was about whose story was correct. Were the captives on La Amistad, native-born Africans, illegally enslaved and transported across the ocean to the Americas? Or were they born in Cuba, where they had no legal rights? This critical decision would determine whether or not the Mendee had the legal authority to use deadly force against their brutal captors in order to free themselves. And the outcome would affect everything, from the United States' relations with Spain and Cuba, to the reelection prospects of President Martin Van Buren and the way we think of John Quincy Adams' legacy to the social division that would soon lead to civil war. Revisiting the film 25 years after its release, it's kind of remarkable.  German Honsu as Joseph Sinke, Matthew McConaughey as Sinke's attorney, Roger Sherman Baldwin, Stellan Skarsgard as the abolitionist, Louis Toppin, who, along with businessman Theodore Jodeson, played by Morgan Freeman, funded the Mendee's defense, plus you had Anthony Hopkins as former President John Quincy Adams, and a very young Anna Pakwin as Queen Isabella of Spain. Major supporting roles are also filled by actors like Nigel Hawthorne, Pete Possilthwaite, Tudel Ejiofor, and John Ortiz. And it features one of John Williams' most moving and sometimes most haunting scores. It's also a masterclass in how to tell a significant story from a deeply shameful part of our history without making its villains cartoonish caricatures or presenting its heroes as pious saints magically infused with the values from 150 years in the future. I can't even imagine how modern filmmakers would treat this story, but I'm pretty sure it would be a preachy identity politics-laden mess. Well, actually it probably couldn't be made at all, but I'm glad it was. And I'm glad it was made when it was because it perfectly encapsulates the most fundamental aspect of what a good film should do. In the words of Pulitzer Prize-winning film critic Roger Ebert. For me, the movies are like a machine that generates empathy. If it's a great movie, it lets you understand a little bit more about what it's like to be a different gender, a different race, a different age, a different economic class, a different nationality, different profession, different hopes, aspirations, dreams and fears. It helps us to identify with the people who are sharing this journey with us. And that to me is the most noble thing that good movies can do. And it's a reason to encourage them to support them and to go to them. This really is the true power of storytelling. And in a lot of ways, I think filmmaking is the most effective way of transporting audiences into a story. But it's rather hard to get anyone to empathize with anything in a story if your main focus is on scolding and insulting your audience. And it's pretty much impossible if your fundamental worldview says that we're all segregated into superficial identity groups that will never understand each other. But that mentality is part of why I'm really concerned about where our society is headed right now. If it wasn't obvious to everyone at this point, we're probably at the beginning of an economic recession. Either way, a lot of what I said a year and a half ago is starting to become undeniable. Even to those who desperately wanted to pretend that they could shudder huge swaths of the global economy without consequence. Back in early 2020, I talked a lot about the damage that stay-at-home orders and chaotic mandates would do to our economy. I've talked extensively about the utter havoc that most of the policies the government has enacted over the last few years would wreak. And while there are a ton of headlines popping up now about how no one saw this coming, you should know that that's a lie. Inflation, supply chain issues, international trade restrictions, volatile markets, rising prices and falling standards of living, all of this was predictable. But because that's not the narrative YouTube wants to promote, this channel took a hit and I'm sure we reached far fewer people than we probably should have. Unfortunately, few people in power seem to be remotely competent or interested enough to do anything that would actually help alleviate the situation. Instead, they've mostly doubled down on almost everything that got us into this mess in the first place. It's not going to go well for anyone and we're all going to need each other to get through what lies ahead. And yet, many of our academic, political and cultural institutions are undermining any hope of coming together as a society. As someone who has had kind of a front-row seat to a lot of the more divisive culture wars stuff that's been brewing lately, it's hard to imagine how we'll fix this problem without many, many more people learning how to effectively and persuasively communicate better ideas to those who need to hear them the most. It's easy to think that everyone who isn't already committed to creating a freer world is an enemy. It can be hard for some of us to imagine why anyone would want a world where property rights are not consistently upheld, where certain types of speech and creativity are off limits, where innovation and entrepreneurship are severely restricted, where somehow in the name of protecting a culture, cultural exchange and adaptation is forbidden, where even the most basic right to self-defense is attacked, where we've concentrated immense power in the hands of politicians and given up control over our own lives. I've spent most of my adult life trying to communicate the value of individual liberty to lots and lots of people. And I've had, I suppose, thousands of conversations with people who either don't understand these ideas, which is totally fine, or with people who outright hate the core concepts, which is baffling. I know what their arguments are. I know how they justify their beliefs, but I still struggle to really grok these kinds of people. Why would anyone want the world, their ideas so obviously create? It seems so miserable. But then when I look back at history, it's obvious that those of us fighting for a freer society have never been the norm. And that brings us back to Amistad. As I said earlier, by 1839, slavery and the transatlantic slave trade was already illegal throughout the British Empire. And abolitionists were already working to change the law in America. But the Civil War was still a couple of decades away. So the crucial thing that the Mende tribesman's attorney, Roger Baldwin, had to prove was that they were in fact from Africa and not from Cuba as their slavers had claimed. And prove it, he did, three times. After Baldwin presents the ships manipulated logs as evidence and the court hears Joseph Sincay's translated testimony, it's clear that the Mende have won their freedom. But with so many politically powerful interests riding on the outcome of the trial, President Martin Van Buren has the presiding judge replaced with one he thinks will be more likely to side with the US attorney. So the slaves have to endure this absurd legal injustice all over again. But in spite of a rigged process, the Mende still win. Were they born in Africa? As such, her Catholic Majesty's claims of ownership have no merit. Neither, of course, do those for salvage made by lieutenants Mead and Gedney. I hereby order the immediate arrest and detention of Seniors Ruiz and Montez. The officials on the charge of slave-treating! Obviously, this should have been more than enough, but thanks to pressure from pro-slavery Senator John C. Calhoun, the case gets appealed again and ends up in front of the Supreme Court. This time, Baldwin, Tappan, and Jodeson enlist the help of former President John Quincy Adams, who at that point was 74 years old and largely dismissed as both past his prime and a poor substitute for his father. But they get to work. In the real case, Adams spoke for four hours in front of the Justices. In the movie, he delivers one of the most powerful appeals for freedom in cinematic history. Our gentlemen, I must say, I differ with the keen minds of the South and with our president, who apparently shares their views, offering that the natural state of mankind is instead, and I know this is a controversial idea, is freedom. It's freedom. And the proof is the length to which a man, a woman, a child will go to regain it once taken. He will break loose his chains. He will decimate his enemies. He will try and try and try against all odds, against all prejudices to get home. It's an incredible speech and there's a lot more of it that I didn't include here. That said, I think Adams' words are only half true. The vast majority of people absolutely do want freedom, but a lot of people also want power. Others are simply weak and do whatever is easiest and most convenient for them, even if that means stomping on other people's lives, liberty and property. This is why I think the battle to protect individual rights for human beings around the world will probably never be one for all time. Freedom is something every generation will have to work hard to maintain. That's why I started this series and in some ways, it's also why I'm moving on. From an early age, I've always been interested in a lot of different kinds of things. Biology, psychology, neuroscience, economics, philosophy, storytelling, art, music and creativity. The more I've thought about it over the years, the more I've realized that the unifying thread that runs across everything is this deep sense of curiosity about why people do the things they do. I think that's where a lot of my fascination with storytelling and filmmaking as a professional craft comes from. As Roger Ebert said, films allow us to explore the world from someone else's perspective. They let us see through another person's eyes and if they're made well, we feel what that person is feeling. And because movies combine the visual arts with narrative storytelling and emotional music, they're extremely powerful in a way that other art forms by themselves can't really approach. Once I realized that I wanted movies to be my career, I went off to grad school for film scoring and then found work in Hollywood, mostly as a music supervisor, music editor, composer and occasionally as a video editor and graphic artist. But working in that industry as someone who had already spent a lot of time reading about economics and political philosophy, I often felt bombarded by ignorance. My colleagues and friends were often driven by the same ideology that has since metastasized and turned into the kinds of insanity we see all the time today. And even before canceling was as common as it is now, it wasn't exactly safe for anyone to speak out against the majority opinion. I've had bosses in the entertainment business who sent out mass emails, literally telling employees how to vote. Conversations at parties and at dinners with other industry people often made it clear that as long as I was in Los Angeles, my career depended on towing the party line. But being a monk and keeping quiet for the rest of my life while the people most responsible for shaping our culture broadcast bad ideas into movie theaters and homes all over the world wasn't really going to work for me. Meanwhile, there was nothing I could point my friends to that they would actually care about that might help them rethink some of their ideas. Let's just say white papers and textbooks aren't exactly persuasive. So I started creating my own videos. Over time, I got better and better at that and eventually found myself where I am today. It took time, but I learned how to become more clear, more concise, more entertaining and hopefully more persuasive. And now I want to take those skills and help build a company that is capable of producing bigger projects, working with a wider array of clients and growing my own creative capabilities even more. So that's why this series is coming to an end. If you've been a fan of Out of Frame at all, I want to sincerely thank you for watching. I've seen a lot of your comments saying that you want this show to keep going because of how unique it is. And I'm really proud of that. But as much as I love Out of Frame, it takes a tremendous amount of my time every week. And what I want to do next is going to require a lot of effort. I need that time to develop new ideas, train a new team and produce new kinds of content. However, that doesn't mean what I've started has to end here. You can pick up where I am leaving off. I wanna see a world where there are way more people making good content in defense of a free society. So allow me to encourage you to stop just watching shows like this and start creating your own. Learn how to write, how to speak, how to debate. Learn how to make videos, how to draw, how to create a great meme. Learn to empathize with an audience. Figure out how to speak their language and give them content that they're actually excited to consume. Learn to become genuinely interested in other people and practice seriously listening to them.  Persuading others to see the value of individual liberty is the only way we can find our way back to a more unified society built on better values. So don't be afraid to start conversations with strangers and remember to keep your cool in the face of disagreement. And lastly, know that it's okay to fail. It's gonna happen, I promise. That's life. As long as you keep getting back up and trying again, your skills will improve over time. No matter what, your creativity is the key to the future of our civilization. So don't just stand by as the enemies of a free and prosperous society dominate our culture. Don't wait for me or anyone else to do it for you. Find something you're passionate about and contribute to the marketplace of ideas. Write a book, make a movie, draw a comic, compose a song, paint a mural, start a podcast, whatever you want to do, it's your turn. Thank you for watching Out of Frame and goodbye.


All right, everyone, let's get started. So thank you so much for attending our talk on custom resource definitions, also known as CRDs. It's great to be here in Shanghai. Let's start off with some introductions. So my name is Sam Gunnaratna. I'm a software engineer working for Pivotal from London. My name's Ed King. I work alongside Sam, also from Pivotal and also from London. Great. So there are a lot of talks about CRDs at this conference. It's sort of a hot topic in the community at the moment. So let's just set the scene about what we'll be talking about in this talk specifically. So Ed will start us off just talking about our motivations for giving this talk. Then since this is a beginner's talk, we'll just cover some basics about the Cates API, custom resource definitions, and operators. Then we'll move on to talking about what it means to build on top of the Kubernetes API for things that are not workloads based on Kubernetes. And we'll talk about the pros, cons, and some of the considerations for using the Kubernetes API for some things that are not so obvious. All right. So I'd like to just take a few moments just to run through some of the motivations that we had for flying all the way out here to China to come and talk to you today. And so I hope that this will help you to understand exactly where we're coming from and also to sort of better frame the rest of this talk. And so this all started about six months ago when Sam and I were assigned to a new team. And we were tasked with building a brand new system component that would eventually integrate into part of a bigger product that we were going to be shipping. And so we went off, we did some research, we spoke to some users, and we eventually came up with an idea for what we might like to build for our alpha. And the question then became, OK, so what technology should we use to build this alpha? Now, traditionally, we probably would have just reached out for a classic web server, probably written in Golang with a backing relational database. And that would have been great. However, at the time, we were very aware of a lot of noise coming out of the Kubernetes community around the Kubernetes API. And specifically with regards to extending the API by a use of these things called CRDs. And so our little ears pricked up, and we thought, OK, what might it look like for us to build the system component on top of the Kubernetes API as opposed to a more traditional web server with a backing relational database? And so we went off again. We did a lot of research. We spoke to some of our colleagues. We took a look at what was going on in the community. And we learned quite a lot of stuff. And so really, the idea for this talk is we just would like to go through some of our learnings and perhaps some of the pros, cons, and considerations for what it might look like to build something on top of the Kubernetes API. So that if you are yourselves asking the same question, like should I be doing this, hopefully by the end of this talk, you will have a better idea as to whether or not that's something that you might to look at further. OK, so let's take a quick step back and talk about the Kubernetes API from a high level. So the Kubernetes API is an API centric. So what do I really mean by this? So when I first came to Kubernetes, I came from thinking about wanting to deploy workloads, wanting to use Kubernetes as an orchestrator and a scheduler for my containers. But the more I came to understand about Kubernetes, the more I realized that one of the reasons that it is such a good scheduler is because it has this API with a set of rich features that allows for the Kubernetes ecosystem to evolve and for features and APIs to be added, an extreme rate from companies all across the world. And it's really the Kubernetes API that facilitates this. So let's have a little look at some of these high level properties now. The first is that the Kubernetes API is declarative. So this is the what versus how approach. So in an imperative procedure way, you might say, I would like to go to node A and start one pod. And then I'd like to go to node B and start another pod. You're telling the system how to achieve something. With a declarative system, you say what you want to happen. So you say, I want two pods. And by the way, don't have those pods run on the same node. And there's been an evolution over time of how you treat infrastructure moving from procedural to declarative.  And then you might have chosen something like Chef or Ansible to manage your infrastructure.  And Kubernetes takes that to the next level being declarative. But also, as you'll see, constantly reconciling that state of the world. So next is that the Kubernetes API has this clear separation of states. So there are two pieces of states. One is the desired state that is specified by a user or another system. Then the other is the observed state of the system. So you have this clear separation that's uniform across the Kubernetes API. And that allows for ease of reasoning about the system. Next up is that the Kubernetes API is level-based rather than edge-based. So what do I mean by that? This is a terminology that comes from system programming and electrical engineering. And it's about how you measure the changes on a wire over time. So in a edge-based system, you notice the individual changes. And you keep a track record of those changes. Level-based, however, you just periodically poll and check the value. And so level-based systems are more forgiving for distributed systems. For example, if your component crashes in an edge-based system when it restores, it has to understand the history of all the changes that have occurred over time. Whereas a level-based system just reads the current value and makes it so. So really, it allows for less error-prone components to be written. And as I said, it's more forgiving. So next up is that the Kubernetes API is transparent by default. So writing resources to the API, you have a single control plane for everything. And permission allowing all of your components can access all of the other resources from other components. So this allows for composability between teams. Controllers can watch other teams resources and then extend the functionality. All right, so those are some of the high-level features that make the Kubernetes API really, really interesting. What I'd like to do now is just dive down a little deeper and walk through a typical example in order to take a look at some of the core components we're going to need to know about in order to understand the rest of the talk. And we're going to go with the classic example of creating a pod. And I'm sure that some of you are probably very familiar with this, so I try to keep it quick. I just want to make sure that we're all on the same page so that we have the terminology to understand the rest. So at its core, the Kubernetes API is a RESTful API backed by an SED data store.  So these are things like your pod, node, deployment. And these are the objects that users of the API are able to interact with and to create, update, and delete. And as a user of the API, it's up to me to declare my desired state of the world. As Sam said, the Kubernetes API is entirely declarative. And so for this example, I might say something like, I want one pod to be running. And the way that you do this is by posting YAML into the API. And typically, this is done with a kube-ctl-apply-f. Let's take a quick look at that YAML. We tried really hard not to get any YAML into this presentation, but that is a really hard thing to do, like when you're talking about the Kubernetes API. So we'll try to keep it brief. The important points to note here are the kind in bold. So here, we're specifying the kind of resource object that we're interested in. And then also the spec, where we are declaring our desired configuration for that object. And so we take this YAML, and we post it into the API. And it goes and gets stored in the SED data store. And so at this point, I need to introduce the next really important component in the Kubernetes system. And that is the controllers. So the Kubernetes API is comprised of a number of controllers. And each controller is watching for any changes to certain objects of a certain type. So for example, this controller in the top left may be looking for any changes to any object of type pod. And when it notices a change, in this case, that a new pod has been created, it gets notified about it. And it then goes through a process that's known as reconciliation. So essentially what happens here is the controller will go and look and grab the desired state of the world. So in this case, the new desired state is that we want one pod to be running. And it will then go and look out to the current state of the world. In this case, the zero pods are running. And it will then take whatever actions are necessary in order to combine those two worlds together such that the desired state is equal to the current observed state, i.e. the one pod is running. And then once this has been done, it will typically talk back to the Kubernetes API to update that original object to update its status. So it might say something like status, state running true, or whatever it might be. And this is really important because this is how other components that are interested in this object are able to find out its current state. So we've got our resources. We've got our controllers. How do custom resource definitions fit into all of this? Well, essentially, custom resource definitions, or CRDs for short, allow users of the API to define their own custom resource types. So perhaps I don't want to create a pod or a deployment. Perhaps I want to create something more exciting. Let's take an example of a lovely piece of cake. Everybody loves cake. What if I wanted to create, update, and delete cakes on my Kubernetes API? This is what CRDs allow us to do. And so in this example, we might say, OK, actually, I want a kind of cake. And in the specification, I'm saying that I want a Victoria sponge type of cake. And the nice thing here is that I can take this declaration, this YAML, and post it into the Kubernetes API in exactly the same way and using the same tooling as for any other resource that would exist on that API. And so I could post that off. It would get stored in the LCD database. But there's now a problem, because by default, none of the default Kubernetes controllers know anything about my cake types. And so at this point, nothing's going to happen. And what this means is that as me as the author of the CRD, I also need to break along my own custom controller and plug this all in. And this new custom controller will be watching for changes to type cake. I will know how to reconcile the world upon any changes to those objects. And this is really at the core of the extensibility of the Kubernetes API using CRDs and your custom controllers. And so you could imagine that you could keep going building up your own custom types, custom controllers, such that it looks a little something like this. OK, so let's have a brief look at the timeline of extensibility in the Kubernetes API. So around, I think, Kubernetes 1.2, third-party resources are introduced. And this is the first time that you had a way to add new things to the Kubernetes API. So shortly after, CoreOS introduced this thing called the operator framework. And so really what they realized is that Kubernetes is great for running workloads, especially like stateless workloads. When things have state and are stateful, you need to concern yourself with the day two operations, like taking a backup, keeping the thing running, making sure things have quorum, that sort of stuff. And so CoreOS developed this framework that allowed you to extend the Kubernetes API to run stateful workloads. And a great example is there will be an operator for MySQL. And so that operator, its job is to replace a human operator that might have traditionally looked after that MySQL cluster. This operator will ensure the cluster is alive and healthy and take backups and that sort of stuff. And this has proved an immensely popular pattern in the Kubernetes community. So the operator framework is one framework of actually doing this, but now there are more, such as KubeBuilder. So in 2017, CRDs were introduced as a replacement for third-party resources to have even closer alignment to the Kubernetes native resources and to be more powerful. And then after that, we started to see bigger systems being built on top of CRDs. So a good example of this is Knative, which is an entire new platform and set of abstractions on top of Kubernetes. Another great example is Istio, which allows you to get an entire service mesh on top of Kubernetes. And I think Istio has something around 23 or so custom resource definitions that it's made up of. So next, this is something that was very interesting. It was that AWS released an operator. And this was special because it didn't act on resources within your cluster. Instead, it acted on resources in AWS's public cloud. So as a developer, I created a custom resource asking for infrastructure, and the operator would go to Amazon's public cloud to provision that. And this is an interesting example of the operator pattern being used to not manage workloads in your cluster but managing something completely different. And so this really got us thinking, like, what is the next step of this transition for CRDs? So just in the near term, CRDs are hoping to go GA in the next couple of releases. There's this movement towards a desire to want to use CRDs to replace some of the native resources in core Kubernetes. So yeah, we wanted to spend the rest of this talk thinking about where this might lead. Cool. So let me paint you a picture. So for the next part of this talk, we're going to be running through an example service and take a look at what it might look like to build something on top of the Kubernetes API. And the example that we're going to go with is that of a home automation service. So we're going to imagine that Sam and I have a hugely successful home automation company. And we've got a really great home automation service that's used and loved by thousands of people all around the world. And before taking a look at what it might look like to build that on top of Kubernetes, let's just imagine what it may look like today. And perhaps it looks something like this. Perhaps we've gone with a classic microservices architecture, whereby some user of the system posts some JSON into some front end, some gateway. And the microservices go off and do whatever they need to do. So let's take an example here of perhaps this is what it might look like to switch on some lights in our kitchen. So we take this JSON, we post it in, and the microservices go off and do their thing. And because we're a hugely successful company, we have hundreds and hundreds of developers co-located all across the globe. And so it's quite likely that each service is going to have a corresponding team who are going to be responsible for developing that service. And really, each team, their main focus is that they want to be delivering the absolute highest quality best lighting service that they possibly can. And so let's draw some team boundaries around each of these so that it looks a little bit like this. And then we can also imagine that each service is going to need some data. And so each service is probably going to need a backing database. And so it's going to look something like this. And then we're going to imagine that we really care about a high availability. And the reliability of our service is really, really important. If this goes down, users are going to be locked out of their home. And it's going to be a disaster. So let's add in a whole bunch of high availability to each of these services. But we know that nothing is ever perfect and things do go wrong. And so when things do go wrong, we're going to need to make sure that each of the services has a whole bunch of retry logic baked in. And then finally, we're probably also going to need some sort of service discovery mechanism so that each of these services can find each other in order to do the job. And so actually, I look at that. And I think that there's a lot going on there. And importantly, there's a lot of stuff going on that actually is distracting each of the teams away from delivering the features on each of the services. And that's really the thing that they want to be doing. So there's a lot of distractions. Each team has to think about their storage, the backing databases, and all of the operational overheads that come with that, the high availability, the reliability, the API contracts between all of the services. All of this is just taking time and effort away from the main focus of each team, which is developing the best service that they possibly can. And so maybe there's a better way. OK. So we very well could have written this service with these microservices. And then Kubernetes would have been a great tool to deploy these services into. We could have created deployments and pods and run this service in Kubernetes. However, given that we now know that the Kubernetes API has all these great features, could we actually build our service on top of the Kubernetes API? So not using Kubernetes as a container scheduler, but just using its API to achieve this sort of business problem that we have here. So let's go through that example now. So first of all, each of the team wouldn't be responsible for creating a microservice, like a RESTful microservice. They'd be responsible for creating a controller instead. And we wouldn't have a gateway. Instead, we'd have our controllers watching a single Kubernetes API. And so before our teams had to manage their own database, now they can rely on the Kubernetes API to store their data via CRDs. So in this world, the user might express intent to the system, declaratively saying, I want the kitchen lights to be on. And so they could post this into the Kubernetes API, and they might have a resource that represents a room, for example. And I have the specification for what I want the desired state of the room to be in terms of the lights here. So you can see I have, I want two lights to be on with different brightnesses. So that could be posted in, and we would have a controller, the team that's responsible for that, watching the API. It would be notified. It would start to reconcile that request. It would look at the system and say, hey, so what would it do? It would create some light resources. And those light resources would represent the individual actions for each light bulb in our room, wherever it is. So the light resource just has a single spec requesting the brightness. And then this would be picked up by our lights controller. And this is the boundary between our two teams in this example. The room team just creates this light resource, and then it's another team's responsibility through its controller to satisfy that request. So the lights controller then would go and look at the actual state of the world, go and talk to the LED bulbs wherever they are, notice that they are not on, turn them on, and then update the status in the Kubernetes API. So you can see here at the bottom, it's set the status to reflect the spec because it has successfully reconciled the desired state, and now the observed state matches. So then our team's controller would be notified of this change because it's apparent of these light resources. It created them. It has an interest of what's going on with them. And then it would update the room resource with the status of the lights. And this is important because the user, or whatever, system component created the room maybe doesn't care so much about the internal resources used to accomplish the request. It cares about this abstraction. And so it can just look at the room resource to see the status of the request. And obviously in this world, if one of our controllers goes down, for example, then it's decoupled from the API. You can still post room resources, post light resources. And then when the controller comes back up again, it can just reconcile. And because of some of the aspects of the API being level-based, that will just work. All right, so that's perhaps fairly a contrived example. I'm not suggesting that you should immediately go out and rewrite a home automation service on top of the Kubernetes API, but maybe I am. Let's see. If you are thinking about doing this sort of thing, there are going to be some pros, cons, and considerations. And so for the rest of this talk, we're just going to be walking through some of the main things that popped up during our research for this. So I'm going to talk through a few of the more purely sort of technical considerations. And then Sam's going to talk through some of the wider considerations around this as well. We're going to kick things off by talking about storage. And really here, I'm referring to this idea that if you are developing on top of the Kubernetes API, you essentially get the backing STD data store for free. It's just there. It's part of that API control plane. And it means that it's probably owned and operated by whoever is responsible for keeping the Kubernetes API up and running. And so as a developer who's developing on top of that API, this is hugely enticing to me, because it means that I no longer need to worry about any of the operational overheads that are associated with managing your own database. But of course, there's no such thing as a free lunch. And there are some other considerations around here. It's not quite that simple. One of the main things that popped up while we were sort of looking at this is that XED is not a relational database, which means that some of the relational features we've come to know and love aren't available. And the example that we tend to give is, what if you needed to do a big join of your data? It's just not really possible. And so I guess the key thing to note here is just make sure that your data model is suitable for running in this new model. Loosely related to the storage is this idea of high availability. So when deployed in a ModiMaster configuration, the Cates API is highly available. And again, that's sort of highly available just for free. And this is, again, really, really enticing to me, because it means I don't have to worry about the availability of my API. It's just highly available for free. And I then have more time to focus on writing my services and shipping the features that I care about. There is another aspect to this idea of high availability. I think Sam touched on it earlier when we were talking about the high availability of the controllers themselves. And so there's maybe a question of like, hey, should I be running my controllers highly available? And I guess the thing to note there again is that in this like level-based system, perhaps it's actually OK if your controller goes down for some period of time, because all that's going to happen when it comes back online is it's going to read the desired state of the world, trigger its reconcile function, and away you go. And the last thing I want to talk about is performance. So again, the performance of your API is going to be largely dependent on the underlying EdCD data store. And there have been some concerns around the performance of EdCD in the past. I guess the key thing here is maybe the key worry is that if you've got a particularly noisy service, that then could potentially have a very negative impact on the performance of not only your system, but also like any other system that is also running on that Kubernetes API. And so we've got to be a little bit careful. I guess the general advice is if you're developing something that's going to be slamming the database quite a lot, like maybe like a metrics thing, or if you're doing something like dumping your logs in the database, probably not a good idea. But we do also have this idea that there's a big conversation going on in the community at the moment as to whether or not you should just be running one huge Kubernetes cluster, or if perhaps it's better to run lots of little ones. And if you're off the mind that lots of little ones is the way to go, then perhaps these sorts of considerations are actually not so bad. It's even possible that you may just have a separate dedicated cluster purely for these like extensions. You may not even enable the schedule on that, for example. So there's lots of things to think about. The final thing I want to say about performance is the API machinery folks are thinking about scaling targets for CRDs themselves. And there's a survey there. If you're interested, please go pull that out. So if we want to take advantage of all of the benefits of Kubernetes being declarative and eventually consistent for our own needs of our own service, then we need to put in some work to actually achieve that. One of those things that we need to do is we can't just take our microservice and turn it into a controller very easily. It's a completely different paradigm of writing software. Your service is probably like a RESTful service, and it's request and response, talking to a database, maybe talking out to some other services through RPC or more REST calls. However, writing a controller is writing a reconciled function that is watching an API and converging state. It's a very different way of thinking and writing software. And as Ed was saying, not everything fits. The Kubernetes API is opinionated and for good reasons. But if what you're trying to model doesn't fit, so you do need that request response synchronicity, you can't map your domain as resources, or you have a very complex domain model, then it's going to be an uphill battle, and you're going to be fighting the API to model it in Kubernetes. Next up, and this is particularly interesting for us, that what we really want to do is collaborate the way the Cates community collaborates. We want CRDs to be the standard interface between our teams. In the microservices example we saw earlier, maybe each team had to define their own API, and maybe you have company standards for that API. We also had to have a service discovery system so that we could discover all the other APIs available. So in Kubernetes, the CRD is the interface between our teams, and what CRDs are available in the cluster is the discovery mechanism. And so this leads to interesting collaboration patterns, where we can have different teams responsible for different CRDs, and maybe independently another team can monitor the status of one team and extend the functionality through this mechanism. So it allows for real loose coupling of interactions between teams. And finally, we don't really have time to go through too many of these, but there are a whole bunch of other API features that you probably want to consider and research if you're thinking about doing something like this. I'll just talk through a couple. So binary data, for example, you should not be using XCD to store binary data. If you do need this, this might not be the right approach for you. Alternatively, you could upload your data to something like S3 and then just refer to it in your specification in your custom resource. Next, if Kubernetes supports defining relationships between resources, and there's two main mechanisms for doing this. The first is a owner reference, which allows you to set up child-parent relationships between your resources.  The other common one is adding labels to your resources. However, it should be noted that managing the integrity of those labels is up to you as the controller writer. So you have to ensure that those labels make sense in a changing system. Next, versioning is interesting. So custom resources allow you to have multiple versions installed at the same time. So you can imagine that teams are writing new versions of their custom resource. But other teams are relying on older versions. So there's some backwards compatibility features there. And now in Kubernetes 1.5, there's a feature that allows you to migrate from one custom resource version to another custom resource version. And finally, since we have this one control plane, the Kubernetes API, and everything is represented as CIDs, we can take advantage of any tooling that exists for this control plane. So kubectl will work, any UIs that are developed will work, the way you list everything in your system will be ubiquitous. So all tooling will be the same. So I guess the question sort of remains, is it a good idea to use CIDs outside of the scheduler? So to CID or not to CID? So if you are wanting to run complex workloads that maybe are stateful on Kubernetes, the operator pattern seems to be the emerging technology and way to do this. And as CIDs get more features, they're going to become even more applicable to other problems, such as the home automation system we showed here. And I think we'll see maybe an explosion of experiments in this way in the next sort of 12 months to sort of see what things can we build with the Kubernetes API itself. But just be careful of jumping on the bam-lugging when any new technology comes out. There's a whole exploration of what is possible. So I hope this talk has sort of helped with some of these considerations and things you should be thinking about. Yeah, and so I think one of the main things that we learn in looking at all of this over the past six months or so is that people have very strong opinions about this, right? Some people think this is a great idea and you should go all in and it's going to be fantastic. Others are more concerned. They just stay away. And so whenever in doubt, we like to look to Twitter to see what people are thinking. There was actually a good tweet the other day, some of this up, which was this one here. And I guess the takeaway is, yeah, this is a new technology. I think people are getting excited about it. And I think quite rightly so. I think there is a lot of potential in this new way of working. And I'm really, really excited to see where it's going to go. But at the end of the day, we need to remember not everything has to be a CRP. And with that, I think that that's all we've got time for. So thank you very much. I don't know if there's time for questions. Perhaps if anyone has any. Or perhaps just come speak to us afterwards. Oh, yes. It's there. That was a very interesting talk. Thank you. So it was a very interesting talk. Thank you. So I have a question. So if I'm making a cluster-wide CRD, and if I want to have only one instance of that CRD on my cluster, is there a pattern to do that? It's like a single-term CRD. Because the answer is I'm not sure. So there's quotas, which have been introduced in the newer version of Kubernetes. But I'm not sure they're cluster-wide quotas. I don't know if anybody knows. I'm seeing shakes of head. So I don't think so. You can set quotas within a namespace, but not cluster-wide. But not today. Any other questions? Is those CRD controller only works deployed on the master node? Or my question is how those controllers scale? Yeah. Yeah, so actually, your controllers don't even need to be running in Kubernetes. So for example, when we're developing these controllers, one pattern we see is that you run your controller on your local developer machine. And you just have to point it and make sure that you have access to the master API. So you can actually run these controllers from anywhere. I think the typical pattern is that you would typically run them inside the cluster. But there's nothing that says that you have to do that. Yes, maybe? So I have a question about the multi-version of your CRD. So not sure in your case, how will you support the multi-version for your CRD? For example, for your case, for team A, he wanted the lighting service version, for example, version one. But the other server team see version two, how you manage the relationship between the multi-versions. The other one is how you validate your CRD schema, how you make sure all the CRD objects is correct. Answering your second question first. So you now have structural steamers for your custom resources. So you can ensure that the custom resource has the correct structure. But then I guess it's up to your controller to do validation. And you can do this via an admission controller so that you can reject requests before storage if it doesn't, it breaks any of your business logic, that sort of stuff. Question again? Ah, OK. So I guess you're always going to have to have communication between teams. So the question was how do you manage having multiple custom resource definition versions and multiple teams that rely on different versions. So I think CRDs allow you to have multiple versions installed. But you still are going to have the problem of needing to know to some degree which teams are consuming that version. Because you don't want to keep your old CRDs around forever as a team. You want to know when you can deprecate them. And I don't think there's any sort of technology solution for that. It's sort of a team collaboration sort of problem. So you mean, for example, I have two versions of live service. So I have no way to force the team A only use version one, but only by the collaboration between team. Is this the case? Using the newer versions in communities, you can change, you can migrate from one version to the other. But the other team has to be aware of that so that you don't break them. Or you have additive CRDs so that they can still rely on the older versions. So I want to know if there are any way to force, for example, the team A only rely on version one, but team B rely on version two. Is there any way to do to force this? Not that I know of in terms of forcing different teams to use different CRD versions. Not that I know of. OK, thank you. All right, well, thank you very much, everybody.


Hello my name is Tim Pearson and I'll be one of your instructors for this course. I've been a technical trainer and a consultant for the past 25 years in both security and in virtualization. The contributing author to the well-received VMware vSphere and virtual infrastructure security securing ESX in the virtual environment. I've been the noted speaker at many industry events including Infosec World 2010, hosting the Virtualization Security Summit, as well as venues like Inatec, GISSA. In addition, I was a named instructor of the year from EC Council for a large pool of their instructors in 2009. In addition, I've also been involved in many military venues including venues at the Pentagon and numerous nuclear facilities addressing both security in the U.S. hair and in Europe. I've also achieved 29 industry technical certification from vendors such as Cisco, Microsoft, Nobel and even EC Council. Lastly, I'm a bi-weekly panellist of the podcast Virtualization Security Roundtable which can be found on TalkShoe.com and iTunes. Hello my name is Dwayne Anderson and I am very pleased to be part of the VMware Advanced Security course. Now I have been able to be in security for quite some time. Back in 2004 I got started in the security industry. I started out actually doing some penetration testing and some forensics work. I've been able to go out and I've been able to test the security infrastructures of many companies and governments across the globe. It's been a lot of fun actually. One of the things that is impressive is that I actually get to break things and I don't have to worry about fixing them. That's a lot of fun. You guys ever done that? You broke it and now you got to fix it? I don't have to fix it. That's a lot of fun. On the forensics side, I've actually even been an expert witness in the courtroom. I wouldn't say that that is a lot of fun. I would rather have not done that, but sometimes you have to do what you have to do. I'm hoping that some of my real-world experience will come to play here as we go through some of this. I'll be sharing some of the chapters here, working through some of this for you guys so that you can not only learn how to implement a lot of security measures with regard to VMware, but you can also do a good job of testing that environment. We want you to be able to see if you can break your own environment so you can make sure that you are secure. So pay attention, enjoy, and I hope to make this a really good learning opportunity for you. In this chapter, we're going to be taking a look at the virtual networking components for the ESX administrator. We'll take a look at the ESX networking components, how virtual Ethernet adapters actually work, how virtual switches work. We're going to compare and contrast a V-switch, which is a virtual switch, which is similar to a physical switch, and how it is actually different from a physical switch. We'll take a look at the spanning tree protocol, and if you're actually running that in your environment, in the virtual environment, you may be actually slowing things down. We'll take a look at the V-switch isolation, virtual ports, uplink ports, network port groups. We'll also take a look at virtual switch correctness, VLANs in our VMware infrastructure. We'll take a quick glance at NIC teaming, how load balancing works, failover configurations. We'll get a 1000 foot view of security features, and then we're going to roll up our sleeves and dive in deep. We'll take a look at managing the virtual network, encryption and certificates, and then finally round it off with Linux file system. We'll discuss the kernel, processes, accounts and groups, and then finally permissions. Now, first off, we need to start off taking a look at the networking components, because the hacker is going to get into your network. Guys, this is where he's going to get in. There are certain key virtual networking components that we want to take a look at. Naturally, we're going to be focusing on things like our virtual NIC, our virtual switch. In addition, VMware, with their vSphere 4 product, has introduced something called distributed virtual switch. Okay, well that's nice, Tim, but it'd be nice to know what it does to understand the problem that it solves. Well, if you have multiple hosts, they all must have the exact named switch in order for things to, well, let's put it this way, flow correctly, the motion to work correctly, things of that nature. With a distributed virtual switch, you don't need to worry about that. It's handled internally in the database and on the system. So that was really, really nice feature that they came out with in vSphere 4. We also have something called a virtual port group. Now, there's actually not really a mapping in the physical world to something like a port group. You could think of this almost as a template. You could think of it as an RJ45 template if that helps, because you can take a certain number of ports, isolate them and say these ports that I've plugged this virtual machine into, this card into, or what have you are going to be used for this particular function. So it really makes it nice for you to group this. It's almost sort of kind of like a VLAN where you do kind of the same thing. Now it's important to understand a little bit of nomenclature before we get started with the course. It's important to understand that this symbol right here denotes a virtual device. You can see this right here is a virtual network card. This is a virtual switch. This is a physical, hence no icon, a physical adapter, all right? Virtual switch, virtual switch, and so on. Now it also needs to be understood that the way we've had this drawn right here, this entire thing is one ESX host. This is a VM, this is a VM, this is a VM, and this is a VM. Now when I typically teach a live class, I'll ask the class, now how many VMs do we have in here? And naturally I'll get an answer somewhere in the neighborhood of, oh we've got four, Tim. Well, yeah, but we have four, we actually have five, all right? The service console is actually considered a VM and not really considered it is a VM. This provides us a lot of flexibility and isolation as well. Now you notice that we have multiple network cards, physical cards coming to one virtual switch here, but you notice there is no uplink port on this one. A virtual switch can have an uplink port, an uplink port is actually defined as connecting to the outside world or it can actually not have one. In this case, the only way of getting access to VM3 is through VM2. So this may be a firewall, it may be a proxy, it may be some kind of filtering device. So I wanted to kind of give you a little bit of an idea of the nomenclature and our environment before we move any further. Now it's important to understand how virtual Ethernet adapters actually work. In reality, there are six different types of adapters. Some of them are used for virtual machines, some of them are used for other purposes. We'll take a quick peek at them and why you might want to choose one over the other. The one that probably has come out latest is this VMXnet, which is a para-virtualized device that only works if VMware tools is installed in the guest operating system. Now we're going to get in a little bit later in the course on some things you may have heard about para-virtualized devices and possibly things that you may have heard about VMware tools. Oh, I don't want to install VMware tools because people have gotten in or broken into the system because VMware tools is installed. Well, I'm going to try and separate the myth from the reality on that. VMXnet 3, its claim to fame is it's built on the same architecture as the VM next, but look at this guys, it provides 10 gigabit per second speed. Now only ESX40 has that and it does require an upgrade to the hardware. Oh my gosh, you're going to have to go back get an order new hardware. Well, you know what the hardware upgrade is? Punch a button. I mean, it's as simple as that. It takes about three, maybe four seconds, okay, maybe five seconds to do a hardware upgrade. The machine typically has to be off to do this upgrade, but it generally is very painless upgrade. We have a VLAN which emulates the AMD adapter and then this is oftentimes referred to as the lowest common denominator. The E1000 generally always works. The reason that VMware chose the E1000 as its adapter of choice is almost every single operating system already has a pre-installed driver for that E1000 card. Then let's move on to the other adapters and check those out real quick. The other adapters, the first one we want to talk about is our window into that service console, which in the graphic previously was our fifth VM. Now we call this the vSwift card. Now the vSwift card is the one that's only used by the ESX service console. In addition, we have our VMKNIC. It's actually a direct connection to the VMWare kernel. It's used for things like vMotion, high-speed connections, NFS, iSCSI clients, all of those types of things. Now it's important to understand that every single one of these adapters, regardless of whether they're a vSwift, a VMKNIC, or the virtual machine adapters, every one of them has their own unique MAC address. They are strictly a Layer 2 Ethernet adapter. For all practical purposes, the operating system in the virtual machine doesn't even know that it's running in a virtual machine. It thinks that it's communicating with a live physical adapter. Now on this slide we're going to take a look at how virtual switches work. Now virtual switches are actually the key networking components. You can create up to 128 virtual switches for each 3.5 host and up to 248 virtual switches for each ESX 4 host. Now the virtual switch is interesting because it's a built to order device. That simply means that as the machine is booted or the switch is created, it actually is created. So we don't have extra code that's running in the system not being used for anything. It does use a Layer 2 forwarding engine. Well the Layer 2 forwarding engine is kind of the key part of the system and actually what gives VMWare a lot of its speed. What it does is it only reads in enough of the header of the Ethernet to make a decision about what it's going to do and then doesn't read anything after that. I kid around in my classes and say, this is kind of like working for a union. Well that's not part of my job. I didn't bid on that particular contract. I don't do that contract. So consequently, this is exactly how I'm going to handle it. So if it helps to think about it like that, then that's fine. VLAN tagging is the notion of adding a particular identifier to an individual frame to know where to go for security reasons. Now we tag a VLAN. Now the switch is also responsible for stripping it off because the VLAN itself, the information, if it arrives at the operating system virtual machine, it's not really going to know what to do with it. It's addressing information you can think of it as. It's the label on the UPS box that you might get. And then naturally we have filtering units that basically keep it away from this area and into this area. We have, in addition to that, our Layer 2 security checksum and segmentation offload units. Now the part that I thought was really cool about VMWare is they made this all modular in nature. They did this so future developments would be very, very easy for them to add things on, kind of like a building blocks as we build a home. All of this dynamically built at runtime, as I mentioned before, utilizing only needed components. This paged the way for us to provide third-party built-in modules. A third-party module or built-in API called VMsafe, and we also mentioned that we have VShield Zoned, are also built into ESX4. Now let's take a real quick look at VMsafe. This is simply an overview of VMsafe, but in reality what VMsafe is, is simply a set of APIs. What is an API? An API is an application programming interface. It's our window into the programmer's world, so to speak. So for example, in VMware's case, when they want to provide some functionality to us or a developer, they'll develop something called an API. We would utilize that API, that application programming interface, in order to build some component to provide communication to and from their particular piece of software.  In addition, it opens up and allows third parties to develop security appliances. Things that the API allows us to have access to are memory and CPU. It provides detailed monitoring of guest VM, memory pages, virtual CPU state, network packet filtering, things like monitoring in and out of the virtual switch. It monitors within a dedicated to security VM. It also allows us to do things like process execution handling. The API that enables complete monitoring and control of the process execution inside the guest OS. In addition to that, one of the really cool things about the VMsafe API is the storage aspects. The VMDK file, which we'll talk about just a little bit later, you can think of as your virtual disk. Now this allows us to take the virtual disk and actually mount it on another storage device. So we would be able to take the virtual disk off of one virtual machine and mount it to say on another virtual machine, almost like passing it around if we needed to grab information off of it or something of that nature. Let's take a real quick look at our current VMsafe partners. You can see they range all the way from ultra networks down to web root. Now there's pretty big names in here as well. As you can see, IBM's in here, RSA is in here, a number of pretty good sized corporations. It's important to understand that we may want to be part of this API or we may decide not to be part of this API. There are certain vendors that chose to become what's referred to as vendor agnostic, which simply means that we can have our product run on multiple vendors. Catbird is a good example of that. Catbird allows us to run on the VMware product, allows us to run on the IBM product, allows us to run on the Microsoft product and allows us to run on the Citrix product. This way it provides them some flexibility, but at the same time they lose some of the really fine-grained details that that API gives you. The virtual switch versus the physical switch. Now let's take a look at this real quick. The SX server virtual switches have a lot of characteristics in common with the physical switches. First off, they maintain a Mac port forwarding table. We look up each max frame, each frame's MAC address when it arrives, and then we make a decision about how we're going to forward that frame to one or more ports for transmission. Now we know that the physical switch, if it doesn't know how to make the decision, it just simply forwards it to anyone. Well in reality the virtual switch doesn't need to do that because it already knows, as I'm going to show you in a little mock-up I'm going to do here in a couple of seconds. It avoids unnecessary deliveries. In other words it's not a hub. Supports VLAN segmentation at the port level and it also can be configured to access a single VLAN as well. I wanted to discuss exactly how a CAM table works. We're going to compare and contrast how it works in the virtual world and then we're going to also do that by understanding how it works exactly in the physical world. Now we all know that every switch that I've crudely drawn over here on the right has a CAM table and the CAM table is its lookup mechanism. We have a port assigned to every one of the devices that are connected to this. The CAM table makes that association by its MAC address. Now I'm going to label my MAC addresses on here A, B, C, and D. I've got very short MAC addresses in this particular environment. By the way, how long are the MAC addresses on Ethernet adapter? Okay, I know I heard somebody out there say 48 bits. That's right 48 bits but in reality it's not 48 bits. It's 48 bits is assigned to the entire card, a portion of it is assigned to the card, another portion is assigned to the serial number on the card. This is what is referred to as the OUI and so I have a portion of my MAC address, the entire MAC address as an example, half of it is the OUI, the other half is the serial number. I typically get the question in my classes, well Tim, is every Ethernet card globally unique and the actual real answer contrary to popular belief is, no it's not. Every Ethernet card doesn't need to be globally unique. The OUI itself is 2 to the 24th power or about 16 million in change. With that said, if it was globally unique, in reality 3Com has only really made 16 million cards. You're probably saying, well 3Com probably has more than one OUI and in reality they do but that would totally mess up my demo so we're not going to mention that. Every one of the manufacturers has a serial number assigned in conjunction with that OUI so they may start with serial number one, run up to 16 million and start over again. If we were to take just simply a router, the packet comes in the router, once it comes in the router the Ethernet header is completely stripped off and thrown away. We readdress that with another Ethernet after we get out of the router. Oftentimes I get the question, well what's the MAC address on that web server, Jim? Okay now think about what you're asking me here. The MAC address changes on every hop through the internet so in reality it's pretty irrelevant what that MAC address is. So we don't need to be globally unique, we only need to be unique on the network segment that we're on. Now if you have more than 16 million cards on one network segment guys, you've got bigger problems anyway. So let's take a look at what might happen in our environment if we very first turn the computer on. Naturally our CAM table that has these associated MAC addresses are going to be completely empty. We're of course going to have our various port numbers but nothing's going to be populated inside of the MAC addresses. Let's assume here for just a second that I'm going to take a frame and I'm going to pass it into the switch. Now naturally the CAM table is empty. Let's assume Al Gore just turned on the internet. He just flipped the switch on the internet. Nobody knows about anything if that's correct English. No one knows about anything. So we've got to pretty much guess about where to send this and that's exactly what the switch does. It does a technique called flooding. Flooding is the notion that it doesn't change anything on the packet. It simply forwards it. I should say frame. It simply forwards it. It forwards it out to every port except the port it came in on. So in this case A would be learned automatically and would be populated in the CAM table. It would then forward it by flooding it to B, to C, and to D. Now B, C, and D, let's assume for just a moment that it's addressed to B. B would be the only one who really responds. C and D would just simply throw it away. Now when B responded, B is populated. Then eventually C is populated and D is populated. At this point in time everyone knows about everything else. In other words all the partners in this switch know about all of the other partners. The convergence switch or the convergence register I should say is set. Thereafter we no longer do this flooding technique. The flooding technique as you recall was sent to everyone. So consequently now and only now is really the only time a switch is secure where we have that one-to-one mapping. Now let's say for example we have an individual we've hired that brings in his iPad or let's say he brings in some device hooks up a hub or a switch in his cube and installs some other devices. Even though we've told him not to, we've now introduced an E and an F into this environment as well. Well naturally the E and F will be populated as part of this large CAM table. Well every switch has a finite number of entries that it can hold. Most switches like the one that we use at home have about 1024 entries. Alright 1024 entries again if you got more than 1024 computers at your house you've got bigger problems in most families but 1024 entries is more than enough. Although their application to a good one by example is something called Mac-Off from Backtrack that gives us the capability to generate thousands of MAC addresses every second thus filling up this CAM table and naturally un-setting this convergence register. Now what happens is we are back to the flooding guys all we need to do is hook up a switch right here excuse me hook up a sniffer right here and simply sniff all the traffic on the entire network or rather everything is attached to that particular switch. So this is how we circumvent a switch and get it to coerce an attacker coerces it to come to him to be able to read all of the traffic on that environment. Now here we're going to be talking about the spanning stream protocol and why it's not needed. We discussed earlier how everything is kind of tied together and we put things together with a CAM table. We also discussed how having another switch introduced into that environment causes problems because things are added to that CAM table we kind of got to move things together. Well if we have more than one switch we need to have one person who decides okay this person goes here and this person goes here. Well in reality in VMware we solve that problem because your switch is a virtual switch. If we go down to one of our favorite electronic stores and purchase something we purchase a switch that's maybe eight ports 24 ports 48 port whatever we're going to purchase if we need to have like say 96 ports and we already have a 48 port we're not going to throw the 48 port away we're simply going to buy another switch and tie the two together. Well in the virtual world you just make a bigger switch remember it's built to order as we discussed before so consequently there is no need to have two switches tied together and the part that controls who's the master and who's the boss in this is a spanning tree protocol and the spanning tree protocol sends around frames to determine this. Well it's not needed all it's doing is just simply creating traffic in your environment that is absolutely useless you know possibly even slowing your system down. There's no need for the spanning tree protocol because you can't hook two switches together because virtual switches can't share the physical ethernet adapter there's not really a way to fool the ethernet adapter and you're doing some kind of a loop back or similar situation they could call some kind of a memory leak or cause a leak between the virtual switches.


Kaden says, what tools are you missing when trying to manage Office 365 or Microsoft Teams? That's a great question. I know that Microsoft is building a fantastic job at solving every problem. Yeah. Microsoft Teams can be mostly managed through the Teams Admin Center. But some things do require PowerShell knowledge, yes, or scripting experience, definitely. What is your biggest IT management task, or what pains do you have when trying to manage your enterprise users for Office 365 or Microsoft Teams? Hey, Jay, before we jump in, I know you have thoughts on that. It's like, how important is PowerShell and scripting in the modern Microsoft 365 world? I mean, it's massive. Microsoft will develop the end-user experience fairly strongly, and the admin experience to a point. But there are so many unique needs from organization to organization that it is easier to give access through PowerShell than it is to develop the admin UI for every single task that's going to have to happen. I will say that the comment, some PowerShell and scripting, I think is a vast understatement when it comes to really in-depth management of your environment. I would also say, so it's funny, SharePoint on-premises, back when SharePoint was its own thing, I always use the 80-20 rule. For Microsoft, it can develop 80 percent of the tool, and then it's up to you to develop the other 20 percent. I think with Office 365, that number has gotten a little bit more in favor of how much Microsoft's developing, 85, 90, whatever. But there's still always going to be stuff that isn't part of the UI. When it comes to admin, reporting is a huge thing. Most of my customers, US Federal, big centralized tenants with tens if not hundreds of thousands of people in them, and many what used to in the SharePoint on-prem days be separated farms now all in one environment. When it comes to delegating the administration for your let's say teams, if you want your contractor for this department to manage all of your teams, there's no good way to do it. You're either making them a teams admin, in which case they can manage all teams, or you're making them an owner of every single team that belongs to your environment, which first kudos on you for figuring out how to do that and not missing any in an environment where anyone can create a team. But second, it's so easy for someone to get removed as owner that to rely on that level of security, it's impossible. Yeah, I could go on all day on this one. On that, I have to say because I come more from the end user perspective, but I have my own tenant, yes, there's that base level, but for someone that's not technical like myself, it becomes an issue because I don't know any of that. I don't know the power scripting, I don't know. It also provides the platform for the partners, like for yourself or Christian in AppPoint, the partners to be able to build on it as a solution, and then those solutions often even get bought by Microsoft and pulled in, as we've seen with Yammer and as we've seen with lots of other things. So it's not like they're not bringing it in or building on it, but for someone like myself or even someone else, it'll also be, well, which solution adds and gives me what I want, and then that can be a harder question to find the right thing to put on top. For me, that's a big issue. It's like, I can't do the technical, who do I rely on then for it? Yeah.  So it's just getting more and more complex, and there's areas which Microsoft is not gonna go and develop around, and that's why you have an ISV ecosystem with solutions. The other side of this too, and I'm just gonna read a little bit beyond what Caden's asking with the question, is that one of the things that's missing is a lot of the adoption and engagement aspects of it. You know, you're wondering, well, I can see that a certain number of chats are happening, a certain number of files are being uploaded, a certain number of meetings are being held, what's actually happening, and you have to go beyond what's in the admin center to really understand what's happening around collaboration and communication and organization, which will have as much, if not more of an impact on the success of your platform than just visibility into all of those things, the tooling. Yep, yep. There's awesome third-party tools that do that too, that do it better than the productivity score. So if you really like granular information about who's doing what, you know, the productivity score gives you a little bit, but it doesn't answer a lot of questions, it can leave some pretty big gaps. Like I've got, next Tuesday, I've got swoop speaking at my user group around all of that and the analytics and how much further it goes and the reports and benchmarking, the global, which is quite interesting on how that actually works. So yeah, I mean, you've got to take it to kind of the next level, but what does that, what does it look like? But they're the big gaps without a doubt, especially on that adoption side. Still, but then that's your favorite insights having come into play, but the cost setting it up and so many people are going, oh, it's all just too hard at the moment. But I'm sure that, you know, because we have as a regular on these sessions is, you know, Norm Young and casually John White, you know, with TIGRAPH. TIGRAPH is great. Swoop, again, you can speak to this, but there's also, there's a reason why Microsoft is rapidly expanding into the Microsoft Viva space and looking at this because they're recognizing that for the long-term, you know, a success of these platforms or these tools that they also need to be looking at those other topics, the adoption, engagement, the health and wellbeing of the employees that use the technology. So it's not always just about those activity stats, but is, you know, what's really happening is the next layer, the deeper layer. That's normal. Well, how many millions of programs have been run out and never been used? It's been millions of that, so, you know. Little things like that. Yeah, that's it, exactly. On the flip side of that, you've got the users going, is Big Brother watching what I'm doing? So, well, you know, they're, yes. Well, the recovering human resource professional, I mean, he's like, going, if you're not, if you don't want to be, no, you're doing it, don't do it work. Yes, 100% yes on that. Christian, bringing this back to the question, what are you missing when you're trying to manage? Something like the second half is question to you, what's the biggest IT management tasks or bans as well? Governance is such a, you know, big one, really. But it's funny. It's a huge gap there. But it's funny, because so for me in federal, there's no money for governance. Everyone complains about sprawl and they complain about the effort it takes to do things, but there's no money for that. There's a lot of money in security. And when you think of governance as a security tool, there's a massive gap in what you can do. For example, the openness that is Microsoft 365. Christian can go create a team. Kirstie, you can go create a SharePoint site collection. Sherry, you can go create a Yammer community. And there is no automation of the policies that are applied to those things. So, you know, Christian, if you create a team and you use that team to talk about employees as part of your legal investigation side of the business, what's to keep me from adding the employee we're talking about into that team? Nothing. And so it's very manual to manage the policy enforcement of all of these workspaces. And it's an issue where we see a lot of scripting and PowerShell because you've got to figure out what's been created. Okay, who owns it? Okay, how is it being used? That's a big area, so automation is a huge thing that while there is a lot of automation capability, it takes a lot of forethought that a lot of people don't want to give. Yeah, and a lot of the training the staff on the security and you've got to then make sure they, and it's very, very rare, very rare. And I'm like, do I do a ton of training? I don't get brought in to do that side of training and helping the users understand some of that best practice around security or asking those questions or so, yeah, you leave a bit of a gap where the security relies on the user rather than on the business. Yep. One area, one big management task is two. And I think it can be, you could put it in either the security conversation depending on where the dollars are for that you need to go and spend to get the work done or over into the adoption side of things but it's around provisioning and automation of the life cycle management of the assets which tie into compliance, which tie into security in each of those different areas. But the, so there are a lot of third-party solutions that help automate, I mean, all different aspects of that but the provisioning of that process which tie into each of those areas. So it really is, as I always say that every new deployment, any collaboration, communication, platform begins as a business analyst activity is to understand what is the culture of the collaboration environment that we're gonna deploy this and what are the tools that are being used? What are the needs, the outcomes for that? And as you're going and designing each of those things, again, there's a lot of it that 80% that Microsoft can do but there's 20% and it's usually the more serious side is the 20%, it's where all the risk really happens that you need to then go, hired guns, go buy tools, buy, bring in experts to help you build out and support those pieces for your organization. Yeah, there's a lot we can dive into on this topic. It really is, yay. There is literally hundreds of hours of webinars on third parties and yeah, yeah. Well, there's just, yeah, there are facets of the admin experience and so it's really just okay, which area do you wanna address? What's your industry? What are your requirements or the outcomes? Nobody likes to work the requirements, I'm old school. I like that word still, I like requirements but yeah, there's a lot to cover. Great question, Kaden. Yeah. If you're a fan of the spinal tap, it's also referred to as trouser marketing.