One of the first tactics mentioned when the executive order first came out was the need to have a blueprint for everything that makes up your software application. This blueprint is called a software bill of materials, often shortened to S-bomb. It shows all of the third-party code and dependencies that together make up the application. GitLab provides a dependency list, which is much the same thing, but the harder part is certifying it. One of GitLab's key channel partners, BoxBoat, has solved the challenge. Cole Kennedy from BoxBoat and Nicole Schwartz from GitLab will together explain more about the ins and outs of S-bombs and how they can help you secure the integrity of your software supply chain. Hello everyone, welcome to my talk, securing the software supply chain with S-bomb and attestation. Let me introduce you to myself. My name is Cole Kennedy. I'm a director of defense initiatives at BoxBoat Technologies. You know, I've been working a lot in supply chain security over the past couple years. Recently, I contributed to the CNCF Secure Supply Chain Best Practices paper. Highly recommend you give that a look. I'm a member of the CNCF Security Tag and the Supply Chain Working Group. I have customers in defense, financing, and critical infrastructure. So I'm here today to talk to you about a problem, right? Right now, software consumers have really no way to really understand the risk level of the software that's running on their systems. And this is because software producers have no real incentive until recently to produce a software bill of materials that gives those consumers of that software understanding of the risk level of that software. And this is a problem because zero trust architecture really requires a strong identity system. And this identity system must really be based upon the immutable attributes of the workloads and the users running on the system. You know, in the recent executive order that I said, the trust we place in our digital infrastructure should be proportional to how trustworthy and transparent infrastructure is. So this is why they are requiring SBOM and software producers must produce the SBOM along with their software artifacts. This gives the transparency software consumers need to understand the risk level of the software on their systems. So the solution, right, is that executive order, right? We have to have some sort of regulation because we are federal, our IT systems and our federal government and our defense systems are at risk if we really don't understand what's in them, right? So the NTIA in this recently defined what critical software is. And the list is very extensive. Anything that deals with authentication access to network resources or network controllers, right, discuss this. I encourage you to go look at the link that I have right here and actually look at all the different requirements or all the different definitions of critical software. And additionally, very recently, the NTIA released what the minimum SBOM elements are, right? And it's important to look at that to make sure that your software is in compliance and your generation of your SBOM. So again, on the executive order on section 4, the timeline is very, very, very fast, right? You know, especially for how fast the government generally moves. By May 13, 2022, there is going to be a report due for pilot programs implementing everything that was in that executive order. So that's within one year of that order coming out. So you go back to the CNCF paper that we contributed to at Box Boat, right? You know, we really pushed for the idea of this out of band verification, understanding that each piece of the software supply chain is actually trustworthy. And once we put all those pieces together, right, we're able to test that the artifact that came out is also trustworthy, right? But we need to explicitly define those trust relationships. And I'm going to show you a little bit about what that looks like in the demo that we have going. But really to understand this demo and why it's so important, I want to step back and explain, you know, zero trust architecture and how to decompose that in a way that really makes sense, right? Zero trust architecture is not a product that you can buy. It's a design decision that you do, that you place upon your networks. At the core of zero trust architecture, I feel is identity. Now, identity is the immutable attributes of either your workloads or users. And those should be cryptographically, they should be identified in either cryptographic tokens such as JWTs or X509 certificates. So everything in your system can be cryptographically verified. The next is policy. Generally, when we talk about policy, we talk about change control boards. And hey, we want to know if a change going into a system meets organizational policy. So we have a group of people that go together review it and see if that works out. But with zero trust architecture, we change how that works. And we actually encode those policies and pieces of code or documents that we can use to make decisions in real time.  And control will take these identity documents from either workloads or users along with those organizational policy documents and put those together to make a decision when a user or resource on the network requests access to compute or network resources. And when you have all these working together, you really do have zero trust architecture, you don't need Kubernetes, you don't need Docker, you don't need any of that. But what you do need is a foundation of identity policy and control. So what happens when we apply these these concepts to a build system? Well, traditionally, right, we have our build steps, and there are chain of steps, and each one relies on the other, right? And if those break down, right, or if there's a compromise somewhere in there, we may never know. And the key to understanding if there is a compromise within that build system is really looking at the metadata inputs and the outputs of each one of those steps, analyzing them to see if they're trustworthy. And if they're actually correct. So without that actual metadata verification, what we're doing is actually placing trust in its producer of the software, rather than the actual software artifact. And that's not good. We saw with some recent attacks from some big companies, right, you know, everyone's really vulnerable to this. If they place all their trust in their software producers, because everyone's going to get hacked. But we need a way to verify that, hey, these build systems are secure. And the artifacts that came out of it, pass this test and the rigor that we expect it to. So MITRE has actually been working on this a little bit in this concept of evidence based trust. This is a graphic from a paper that was released earlier this year, called Deliver Uncompromised. Now, if you go look it up, there's two of these papers, right? There's an older one or the newer one. I recommend that you look at the newer one. Because it really builds on this SBOM and this evidence using the evidence actually make decision execution time, right? So the concept here is that we bubble up all that metadata from the different stages of your CI pipeline, you sign that metadata. And then you can use that metadata downstream to actually make decisions when that resource requests access to network resources in the form of certificates with an identity or it requests access to compute resources, right? Add admission control. So let's decompose that, right? Each of those CI steps has different inputs and outputs. And when we look at these inputs and outputs and we can build policy against them, we can have a lot more trust in the system. So different things that go into a CI step, right? You get your runtime environment, it might be what's your Docker image that you're using it. Then you have your source package, right? You have your internal code that you're compiling. And then you also have packages that you may be bringing in externally. You have build tooling, you have compilers, linters, packages, right? You have all these different things. And you also have machine identity, right? Where is that actually being built on? Is that being built on a server in a basement somewhere in Russia or is it being built on your hardened infrastructure in your AWS security group, right? Those are things that we really want to know about. And then, you know, the outputs of that CI step, right, are of course, you know, your artifacts, you're going to have your S-bomb, you'll have different test results, you have some logs, and then maybe you have some traces or sys calls so you can really understand, you know, what happened at the kernel level during that CI step. So when you can take that CI step metadata, you push it to an external store, you make that available, right, to a metadata verifier or some sort of a mission control, that allows you to have a really strong identity system over your software artifacts, right? And this is really what the NTIA is looking at with the S-bomb, right? With that S-bomb, we push the S-bomb to an external store, and we're able to get insight into exactly what is in that software artifact and make real-time decisions of whether we're going to allow that artifact to have access to network resources and compute resources. So, you know, we've been putting together, we've been doing some work on Intoto to actually make this happen. Intoto is a great open-source piece of software. What it does is it allows you to sign metadata from each of the build stages in your CI process, and they compare those pieces of metadata to a build policy that you create. And so that build policy then verifies that those steps actually took place as you expected them to. You'll see a little more of this when we get to the demo. I'll make a little bit of sense. But the key part of this is that it really decouples verification from the build steps. And almost every single CI system that I've seen, right, that verification actually happens as part of the CI process, right? So, you're almost forced to place that trust into the software producer, right? Now, within Intoto, in this pattern, the software consumer is actually able to do that. The other piece of software that we used was Spire. Spire is a system for distributing certificates across infrastructure. It's got a robust plug-in system, right? So, you know, we actually wrote a plug-in that ties into a hardware road trust using a TPM 2.0. Works absolutely great. I really love the plug-in API. They have a great community. And, you know, it's a standard-based, it's standard-based, which means there's a government body to it. So, it's not just going to change whenever, right? They actually want multiple implementations of the Spiffy, of Spiffy. And Spire is just a reference implementation. So, when we take those two projects and we put them together, right, we have this automatic way that we can deliver certificates to our workloads using Spire. And then we also have Intoto, which takes the inputs and the outputs of a build step. And it allows us to verify that that happened. That allows us to actually build an automated system that satisfies those goals that that MITRE had. So, I'm going to switch over to a demo here and kind of show you how everything works.  So, this is our build policy, right? And this is completely decoupled from our actual build steps, right? So, what we do is we sign our layout that validates that our build step happened as we were expected. And we do this with an offline key. In this demo, we're kind of faking it a little bit. And we have our key just sitting in the repository. As you can see right here, it's his owner key. And I have a little script that actually signs that layout file, takes that unsigned layout and then signs it, right? And then we verify that that signature is actually valid at the verification time. So, then let me go into the GitLab CI and kind of show you exactly how we're wrapping our commands with Intoto. So, right here you see we have the Intoto command. Now, this Intoto is from our fork. This is not the upstream Intoto. We're really working hard to get this upstream. There's a few changes that we need to make. Hopefully, we'll have a PR in there by the time you all see this. So, we have Intoto, then we run and then we're cloning where the name of the step is cloned and we're specifying some products, right? So, the output of this is going to be a directory named go hello world, right? And that's expected. And then we have our Spiffy workload API path. Now, what we did is we exposed, we deployed Spiffy on the cluster that GitLab is running on, and then it mounted this socket into each one of the build containers. And then we actually just have our command right there, right? So, we're actually doing our clone command and then making sure that we're at the right SHA hash, right? And so, the artifacts for this are going to be something called a link metadata file, right? And this will contain all the information about what happened, the products that are the materials that went into this, and the products that came out. So, and then let's go down to the next one, right? So, the next command, right? We're doing our Intoto run again and this is our build step, right? So, our materials are going to be everything that's in that directory. That's everything that came from that clone step. And then our products, right? We're building it, so it's going to be a binary that comes out. And then what's cool here is we're actually creating an spdx, which is that sbomb that we talked about a little bit earlier than talk. So, we have our Spiffy workload path again. And then we have our go build command. And then we're using this bomb tool from the Kubernetes distribution. So, there's a recent PR where Kubernetes is actually creating an spdx sbomb. So, I pulled that tool and added it to that base container and generating that sbomb right there. So, that should work, all right? So, then let's look at the next line here. So, now we're building an image, right? And we're creating a tar file docker image. And then we're doing a scan. We're using Trivy. We're using the latest image there. And then we have our script right here that, hey, downloads Trivy, gets everything ready to go. And that, hey, again, you see here where our products are the Trivy scanning report and our materials are that gohelloworld.tar. So, we're telling them we tell Trivy, hey, go take that helloworld.tar and create a scanning report and that JSON. And now we have all these different artifacts. So, we see we bring those out and those are going to be available at the end. And then the last step, this is a verify step. Now, in this pipeline, in this example, we actually have that verify right before we push it to our registry. Ideally, in a zero trust architecture, right, we want to actually verify it before we give access to network resources or compute resources, right? So, this might be at admission that we'd start running some of this stuff. And, you know, that's some work that we still have to do there. But you can see we have this in total verify. We give it that sign layout file that I was talking about. And then we give it the public key for that layout. We can also give it X509 certificates too.  And then you really have strong control over what goes into that system. So, you can see, and then we also connect to the specifically workload API so we can grab that trust bundle, right? So, then if that gives us a good exit status, then we push it to our registry. So, let's go ahead and kind of actually see what this looks like. So, this is a recorded talk. So, kind of cheated a little bit. I know this is going to work. If my internet will get fast. All right. So, let's go look at that one. Don't look at that failed one. We'll talk about that one in just one second. So, right, we have this clone step. And you'll see artifact. We have this link metadata file. All right. So, now we have, you can zoom in here. It might be a little bit tough for you something to watch. So, you can see in this link metadata file, we have the products, right? Hashes of everything in there. And then we have byproducts, right? What was our return value? What happened on the command line? What all that happened? You know, and then it has our command. So, everything here and then you can see we actually have a signature and then we have our certificate here, right? This is some of the work that we did actually with the certificate constraint system to make that happen. All right. Let's close that out. And let's go into the next job. So, you see we have that link metadata. I want to actually open that up. But we also generated an spdx file. So, let's go ahead and download that and actually see what that looks like. There we go. All right. So, look at this. We have a compliant spdx file, right? And then if we actually go and we look into that link metadata file, let's download this guy. So, you have all our material. So, we know what went into that sbomb. And then our products, right? There we go. We have our hash right there for our spdx file. And we have our signature over this whole file. So, we can actually validate that that's correct. And we do that. So, let's go back into our verification step, right? And we can see right here that, right? There we go. In total verify. And we have some mornings, right? I think we can we can lock this down a little more if we want. But, you know, all the files, all the inputs and all the outputs actually pass. And we'll go actually take a look at this layout file one more time to show you actually what we're doing there. So, we're doing actually a verify and we're doing an inspection. We're making sure that our go-hell world is made with products from the build image stage. We're matching the image ID, right? That's a hash of the image and we're making sure that's made with products from the build image. Same with the trivia scanning report from the scan image and then that go-hell world spdx with products from build. So, right here, we're actually doing cryptographic verification that everything that came out was from those materials that we put in that was from our Git repository. We can extend this even a step further, you know, and verify that all those commits were signed by developers if that's within our organizational policy. And then we're checking here, right? So, our cert constraints. This is the work that, you know, we really spent a lot of time at Boxboot was because traditionally in Toto, it requires, you know, public key, private key pair, right? That doesn't really work with enterprise organizations or using short-lived keys. So, we added a certificate constraint system that allows us to specify fields on that certificate to see if that step was valid. So, during this step, we used a container that we gave the identity of in Toto builder to and we're making sure that, hey, yes, that container was the one that we expected to. So, if a malicious actor replaced that container with something that they put in there that might have some of their malicious tooling in it, right? It would never be issued this identity and we would not pass this step. So, you know, and let's go take a look at what that looks like to what happens if we do do something.  So, let's go look at the changes they did. Look at this, right? So, they did a said in our GitLab CI.yaml, right? What developer looks at their GitLab CI.yaml after they already got it set up? So, a malicious actor could have put some sort of trojan horse in here, but because we're using in Toto that verify actually failed. Go look the logs for that, right? Inspection failed. Air artifact verification failed. Recept failed. Materials main.go disallowed by real disallowed. So, right there. We caught the hacker and now I think that's the end of the demo. So, I'm going to hand it over to Nicole and she's going to go over how this applies to GitLab. Thank you, Cole. So, what does GitLab think about this? We love that Boxbo did this proof of concept, leveraging the open source tools in Toto Inspire to incorporate attestation documentation into a pipeline. It weaves from multiple GitLab stages, manage compliance, verify, package, release, and secure. They're showing off the power of all in one DevSecOps. We at GitLab have also had an increase in customer interest in software bill of materials, signed build documentation, and many more complex verifications within the build process. As Cole mentioned, we believe this mostly has to do with the executive order here in the states. However, there are many regulated industries who want these capabilities. GitLab is watching the Linux foundation, digital bill of materials, and other projects with the desire to integrate them into our pipelines. Now, here are some roadmap items. If his demo peaked your interest, you may want to keep an eye on these. Verify build artifacts, package release evidence, release compliance tooling, and secure composition analysis software bill of materials. And here are some of the relevant GitLab backlog issues. The best way to make sure that the relevant product manager is alerted is to upvote and comment with any additional context on why or how you use the proposed feature in the issue so that they can be sure that what they've designed is really what it is that you want. You, of course, are not limited to just showing your interest on existing items in the GitLab backlog. You can propose new features if those don't cover your needs. If you're a developer, you could actually go ahead, take the proof of concept from the BoxBoat blog, and start incorporating that into different pieces of GitLab and move to get those relevant pieces into GitLab itself. Remember, everyone can contribute. Or if you would prefer and you're a BoxBoat customer, you should chat with your account team and let them know that this is something you would like to invest in maturing. Thank you for coming to listen to us talk about securing the software supply chain with S Balm and attestation. We look forward to your questions. Please enjoy the rest of commit.


Saluton al chioj, eni vu diya viya diyo. Ho diya o estas feria tago chitia en sydneyo, sed ho diya o mi intensas laburi chela Esperanto domo kund Dmitri de nove, nidaurigos nian reno vigon de la domo, kai ankao mi keretis ke kaya venos, li didis ke li venos, sed niviros, char li ofte shanjasi an decido nia la lasta momento. Mi aishu oi ankorau estas malkurai pro la lasta tempokia mi estas chela Esperanto domo. Do pasint merkerede mi fareis interviuon chela Esperanto domo kund alia esparantistoi. Aprecipe pri Esperanto ser ankao mi parolis iu metepri amikumu. La aritikolo kund el sendo jus estas publikigita, mi emetos ligilo nla periscrivon eble vii shatos alskutigin. Anka vii shatos al pato, ke oni didis ke mi estas tutmonde respektata YouTube-isto en la Esperanto komunubo. Saluton kya li vi? Buone, do mi suposis ke nii bezona chytiun ilo in hodiau. Do chu vii shi oi memodas ke antao estis ponido chyti, ke nund tuta nia estas. Do hodia nii simpe intensas farbi chytiun, la tutan muron, ke nund mi iras al agast ciambro kye dimajiam komensas labori. Ke nii komparaneble nii debos farbi chytiun, sed shainas fek, vi preska nia paus vidi kiel Ma supono de la podido estis antao e eble pro la lumeo kye di pous vidi nund, o mon momenton. La podido nund asvektas kiel podido, neplu kiel trua e lemura kam tia othe, do nii farbos ke kiel nallian nii faros dimitri. Ni shanjis la directo oi ordo. Oh ye ye, nii debis shanji la directo, yes? Ne, nii shanjis nii nebizonas farbi kiel  Perquè సానీన, సానీన సానీనుపేూ వివిమీలువర్విసానేతూ సంవీగిం నేప్ట్మూ సింమూరోంర్సాను సింవానును. సింవిసాను సురోనోను. So, we put all of this in the boards And in English we call it floating flow I don't know at all how Esperanto is So it's just boards Yes, in fact, we need to create the word Yes, yes So maybe it's the floating board But that's a strange idea because I like that it's actually floating and moving when you're walking around it No, it's not a good idea We don't need that as a nail Yes, yes, yes We don't need nails Yes, and it can also move when the temperature changes Ah, yes, yes So it can enlarge it and not enlarge it depending on the temperature So we just drank our tea and ate our biscuits and now we intend to start working and I have to light this wall to perfectly light it before we light it So I'll do that now So I just finished lighting the wall here and now I just intend maybe you can see here I have to use I don't know how to name it in fact, I just show it's so many of them Look, how it's finally coming here I always film everything I do well, leave it to me but it doesn't matter And finally, in Venice it's maybe two hours late but only two hours Only two hours and two but at least it came during the right day that's always a good thing So I guess now I have to light it so let's do something I hope that these rooms are not going to be more luxurious and good because after now it's not going to be too late Yes, maybe You have to wait for Dmitri and he will tell you what you have to do So the plan now is for that that it's here and I have to plan the walls here between the doors and between the wall and the wall and that and after I have to plan that and after that me and Kaya will work to make the wall we won't work because I need help I just have to look at him for sure to make sure that he doesn't do something stupid I don't do anything stupid Why am I stupid? Why? Always The sub-bar is already done and now we have to wait two hours and then we can make the sub-bar and now I will understand what others will do I like to play What do you do? Why do you do that? Why do you have to do that? A credit generation Yes Is that... No, that's a cat I don't know So now it's time to eat and the rest of this we just have to while me and Kaya and me and Lidus will buy food because I think you already bought it So after we'll just eat discuss stupid things in Esperanto You really have to I don't have a finger as I can say In fact there is oil because when I did a week I need to use oil because that made nothing I just now put oil for Kaya So it's not so I suppose they just have to eat with with dirty hands So we just finished eating and now we start to work again as you can see Dimas is there with Kaya and they just eat like it's a good worker because Kaya is the most poor worker ever but good So this is my very good job In fact Kaya did this on purpose Here's everything he did on purpose and it seems that we were in the ambush because we had to do that So now it's night Kaya is lazy and we don't want to do this So I just want to do some things and Kaya is attacking Mosquitos In fact again So I show what we did So Dmitri did this very good job Dmitri and also we did the candles here and I did the chamber Also we did also we did outside but I can't do that now because it's very bad So Hey, let me tell you a joke Okay, good Let's do it How lazy are the workers because they have workers workers Do you understand? In fact I don't have any money So now we are at the end of the film I'm lazy I'm at home So if you like this film you know what you have to do like it share it subscribe to my channel if you still don't like it and I'll see you in the next film and if you're not there I'll come to your house with Kaya and I'll put her candle in your face


Good morning, everyone. Thank you very much for coming to this roundtable. If you've not read the schedule correctly, the subject will be neutron overlay or SDN. And the question is about the choices of networking models that neutron offers. And it's not only a vendor problem when you're choosing that, but it's also a question of how you're going to set it up. And one of the questions that one can ask is whether your tenants should be able to touch your physical network or whether this should be limited to accessing an overlay. Another question is what are the limits of these models? Well, I've asked five very fine gentlemen to come help me ask this answer this question. I have with me Mike Cohen. Mike Cohen is director of product management at Cisco. I also have Dan Dimitriou, who is the CEO of Midokora. I have Pedro Marquez, who works at Juniper as a software engineer. And finally Chris Wright from Red Hat, who is head of the CTO office. And I think I miss you, David. I don't know why I did that. So I also have David Dynrow, distinguished engineer at HB. So the rule of the game is that each of them is going to be giving a five minute statement to start with. And then we'll open the floor to questions either from the people on the roundtable or from the room. Thank you very much. Mike, do you want to start? Yeah, thanks a lot. So this is a really interesting question. And it's one that we've been thinking about with, you know, at my group quite a bit. And ultimately, one of the conclusions we came to has been driving some of the work we've been doing at OpenStack. And really, if you start thinking about how tenants want to consume OpenStack, there's multiple classes of users. And really, it's about how you want to launch and create applications and making that process as fast and easy as possible. And ultimately, if we look at the way we've been, you know, a lot of OpenStack projects have been created to date, you know, the goal has been to create an abstraction layer across the underlying hardware, and then allow that abstraction to be things that tenants access directly, expose a set of tenant APIs. You know, but the reality is is, you know, as we looked across the different projects, we really saw that, you know, many of these constructs that we're exposing are still very low level. We saw some of these things with overlays and you know, in virtualization layers.  You know, this is still relatively complex in our minds.  So the tenant experience is more complex than it needs to be, because a lot of the low level details of managing, essentially managing the infrastructure still bleed through in the APIs that we're exposing. So the mindset that, you know, we've had, or the way I've been thinking about this problem, is about you really capturing user intent and essentially how you model applications and be able to model an application separately, completely separately from the underlying infrastructure. So I guess probably in, you know, in, you know, in Xfranacular, this may feel more in kind of the full SDN. But really, it's separate, you know, it's to some degree a bit separate from these underlying models. The idea is, you know, we need to, you know, we need application developers or the underlying kind of end users, you know, in the tenant to describe what they need in an abstract way from the cloud environment. And then be able to translate that through the system to map that to the underlying resources. So this gives us the best of both worlds in a way in that it actually exposes a lot of power to the tenant and that it allows them to fully describe the environment they want. But we don't want them to do it in a way that is directly tied to the low level constructs. Because then we actually have very limited flexibility about what we do. If they ask for something low level, we have to give that to them. That's what they asked for. And we're honoring that contract and automating it. If we, if they ask for something abstract, then we actually have a lot of flexibility or the underlying system has a lot of flexibility and how that can be managed and deployed. So as we've been thinking about how, you know, how opens that can evolve, we've been thinking about these kind of models. You know, part of this is present in the group based policy project, you know, that we've been working with the nutrient community and also the broader opens that community around so far. So you can you check out that project to see a little bit more of your sort of this this concept in this architecture. You know, the other idea that's really important and heard in this is that you need to separate concerns. So as you give tenants a very simple application model by which to, you know, by which to describe what they need from OpenStack, you also want to have a separate input, you know, a separate place by which providers can, you know, actually describe on the underlying system or handle the operational aspects of running the cloud. These two systems need to be separate. They need to take inputs from different places. So as we think about the models we want to expose, we want to be able to allow the provider to set out, create a set of underlying rules that can be merged with the abstraction models that we've offered the tenants, and that can actually be put together to program the underlying hardware. So again, this is another case for adding the layer of abstraction and not tying the very low level constructs all the way through to the tenant. Because if they ask for something, you essentially have to give it to them. But you know, then it makes it very hard to marry it together with the request of the provider and ultimately control the system. And as we've seen with, you know, some of the other work that's gone on is you layer on things like governance and compliance constraints that even adds more complexity. Because if your low level details bleed all the way through to your tenants and they say ask for something they're not allowed to do, you either have to fail this in different ways or disable something they've already asked for. You want to separate out these different requests and give them a very simple API, but one that actually has the full underlying power of the hardware underneath. So at a high level, that's the way we've been thinking about this problem. You know, how this approaches SDN, there's obviously a number of ways to, you know, to approach the implementation of your overlays, you know, and how overlays can be integrated with physical hardware in different ways to achieve these kind of solutions. And with Cisco, we've been working on a range of these things. People may have checked out our application-centric infrastructure, for example, which actually ties together an overlay with the underlying hardware to give you essentially a system that can tie across hardware platforms and software and give you integrated visibility. But at that layer, there's a lot of different approaches by which to achieve things. The key concept is actually the separation of intent from the underlying system that I'd love everyone to take away. Think on that, I can hand off back to Nick. Thanks a lot, Mike. Dan, do you want to continue on? Great, thank you. Good morning. Mike, why don't you go and sit up in the front? We don't have chairs, so we'll make do. So I pretty much agree with everything Mike said. But honestly, I think the question was, initially, should we allow the tenants to have any kind of direct access or control over the resources that are provisioned in the physical layer when they want to provision their applications and their virtual network constructs? And I think we pretty much agree at this stage that the answer is no, not directly. And there are many reasons for this, like Mike already mentioned, you know, I'm going to repeat several of them. We don't want to tie the users and the applications to specific hardware. We want to abstract them away so that we can change that underlying hardware model at some point or optimize it. And, you know, we want to actually get away from some of the limitations of the underlying networks as well. So one of the reasons why we go for the overlay approach is because we try to overcome some of the existing, perhaps still existing, limitations of the hardware models, right, like the number of isolated network segments that we can effectively create and such, right? So perhaps some of those, you know, are resolvable in other ways, but we have to do that in software, right? There was a good point made to about allowing the administrators, the operators of the cloud, to set up the infrastructure and control the things that tenants don't really have access to, but that are important, like, for example, provisioning connectivity to external networks, either to the internet or to, you know, private corporate networks. And I think that this is one area where currently Neutron actually is falling short. So from, for creating a tenant-facing API, you know, for provisioning network segments and routers and other network constructs for application developer owners, it does okay. Of course, we do need to layer the policy of stuff on top of that so that it's even more abstract or so that people don't need to care about the application, people don't need to care about the low-level details, like IP address management and firewall rules and things like that. But on the other side, we're not doing a great job of expressing realistic networks for the administrator, because practically everything's modeled like a layer two segment right now. You know, and I think realistically, a lot of networks have more complex scenarios, like, you know, of course they have routing to the internet, peerings with BGP networks, sorry, MPLS networks, various types of carrier ethernet and such. And none of those are actually modeled in Neutron Raw. So it's actually something we're addressing, at least on the layer three side, I'm going to make up cheap plug for one of our blueprints, you know, the provider router blueprint, and it's associated IPAM changes. Please check that out. So it's a small point, you know, but it's something where we think that for the administrator, we really need to do a better job of providing the capabilities, right? So, you know, overall, getting back to the original question, so the tenants can't directly modify or affect the underlying networking, right? But indirectly and through various policy mechanisms that are in place, they do. So for example, when we're provisioning networks, currently we just kind of assume that we go over the top as an overlay on IP, and we don't pay a lot of attention, you know, to what's going on on the underlying network. Realistically, there's going to be, there's going to be a congestion, resource limitations, and we have to take that into account. We may have to actually do things like rerouting traffic, you know, paying attention to priorities. Currently, we don't expose priorities to tenants, right? So we probably should do that at some point in some form. Not sure exactly what that is, but I think that's important because it comes up frequently, like people say, well, my application, A, B, and C, are very, very important and, you know, some of these other ones are just best effort. We have no way of expressing that in a standard way, let's say. Maybe some of our products individually do, but we don't have that in the base. Right. So it's a lot of things of this sort. I didn't talk much about the policy, because, you know, Mike already talked about that, but I definitely agree that the more abstract, declarative policy-based mechanism for describing networking requirements for the applications is ultimately the way to go, right? So whatever we all decide on and gets de facto standardized, you know, I think is is going to be good. And then it's up to us to implement that in an efficient way on the on the other line network. Thank you. Thanks a lot, Dan. Do you want to join us on these wonderful shares we have? So, David, your turn. Thank you. Morning. I'm a I'm a big SDN guy. So the answer is full SDN. Thank you very much. No, I'm kidding. I think in the SDN world, we're building a more nuanced sense of what a tenant actually is. And that, you know, I do believe we should expose full SDN in OpenStack because it's going to bring some richness and some new capabilities, including sort of a hierarchical notion of what a tenant is. You know, one person's provider is another person's tenant, and you could end up with a scenario where you have facilities based provider that owns physical resources, switches and routers and things of that nature. You could have multiple virtual providers that had resources carved out from the physical provider's space and allocated to it that are under the control of that virtual provider. They're going to have corporate tenants that are allocated certain resources and certain things they're allowed to do in the network within the corporations, you know, business units and in the business units, administrators and etc. So the notion of should a tenant be able to do dangerous stuff, I think we need to think about it more in terms of which types of tenants should be allowed to do dangerous stuff and which shouldn't clearly at the top of the stack, it makes sense to have a completely abstracted model where nobody can hurt themselves and they don't have to know anything about networking, you know, completely agree with everything that Dan and Mike said and that's kind of the beauty of open source is we work for different companies, we compete with each other. But in this world, we're just talking about reality and what makes sense and we're all drawing the same conclusions. So I think there are people who should be exposed to the low level protocol details and the vendor specific and the media details. And I don't think it's as simple as one's an administrator and the others a tenant, but that it's spread across the hierarchy and that you can use policy to define what it is that your child or your tenant inherits from the space in which you live. So I'm hoping to see OpenStack and Neutron expose more of the richness of control over resources that SDN offers, you know, a range of abstractions and the appropriate abstractions pointed at the appropriate system users. Thank you. Thanks a lot, David. Pedro, your turn. Good morning. So I mean, I think I'm taking the question a bit as the question of network design. And to me, network design is about requirements. So when I look at the cloud space, in my opinion, we're going to see a smaller number of very large clouds. And if you look at network design requirements, I think for a lot of these large clouds, you're looking at, you know, network requirements of, say, 10,000 to 100,000 ports, non-blocking, full 40 gig. We know of our subscription as an example. And within the industry, we kind of know how to make 100,000 port network work. You build it as a clause designed and you will end up with something like four or five thousand switches, you know, at that point, it becomes really important to cost optimize those switches and to limit the function of those switches. So and, you know, it's in a clause network, really, VLANs are not possible at the underlying. It's just not possible. Multicast is not possible. So both, you know, VLANs and multicast based VXLAN are not, you know, feasible technologies. So if you take that point, which I think it's relevant for some very large designs, I think there's at the very least for very large scale networks an absolute need to build the overlay if you're going to implement the semantics of the Neutron API. So if you're a cloud provider that implements the Neutron API and if you allow your tenants to bring in any address space and implement sort of what Amazon calls the VPC API, which is what Neutron does, there's no other practical way to do it other than an overlay, in my opinion, at that large scale. There's also at that scale, I think a lot of those operators really want to drive the design to simplify the requirements of those devices, such as those devices can be part of the word, but commoditized and commoditized, not just to drive the on price, but to drive a standard of operations.  If you have a small cluster of, say, you know, a few racks, the point that I made previously does not apply. So that's kind of one topic that I look at. The second thing I look at is, when you, at least for me, when I think about networking, I don't particularly think about L2 segments. To me, networking is about routing and very commonly in terms of private clouds, which I tend to work with from a manageability perspective, people want to deploy different tiers of applications into different virtual networks because they are different administration domains. And, you know, like in your traditional cloud application, web servers don't really talk to each other. They just talk to the caching layer. The caches don't talk to each other and they just talk to maybe a database or some storage. So traffic tends to flow from external, front end, cash tier, app tier, database tier, and never side-to-side traffic on that L2 segment. So in, if you're, you know, logical assumption is that pretty much your L2 segment is irrelevant. What is relevant is what you do about routing. And I think in a lot of discussions we're having in terms of OpenStack, we're kind of assuming, well, routing is this thing on the side. It's either, you know, a completely external system or it's this L3 agent, which is Linux VM, with some rules being pushed into it. I mean, the approach we've been taking from the beginning, and I think if you look at what's happening in terms of the distributed virtual router work in Neutron, I think the, you know, the community is really going in that direction is that the key thing is what happens to routing?  Has it's traversing layers? And, or can you do distributed routing? And how can you do interoperability between a Neutron network and some other network virtualization technology outside? So I mean, taking the needs of large scale, taking routing as the primary factor, you know, the approach we've been going down is built on using an overlay with fully distributed routing and interoperability with existing network-based network virtualization technology.  I'm sure that there's many other valid network designs if you take a different set of assumptions. I mean, it's, you know, with different assumptions, you end up with a very different logic. Thank you. Thank you very much, Pedro. Chris. All right. Hopefully, it's on. The benefit of going last is really, I, I, yeah, there's still people out there, okay? Benefit of going last is everybody said some variation of what I would like to say. So maybe I'll try to change the pace a little bit. Some, some key pieces that I think we need to reflect on are what are we trying to achieve with giving users access to the network in a cloud? And I think that Neutron right now is fundamentally failing us. It's presenting to the users as Mike started off this discussion. It's presenting to the users a fairly low level set of network engineering focused APIs. And then, as Pedro mentioned, what is the value of a Layer 2 segment in this environment? So we've built a system whose primitive is give me a Layer 2 segment. And I think that that's something that we really need to address to help move the whole state of the art forward. Definitely decoupling the tenant notion of how they manage the network from the underlying physical infrastructure is important. And I think everybody's touched on that. So call that an overlay. There's a lot of different technologies you could use to build an overlay. Some of them are common to the modern SDN, definition of SDN. Some of them have existed for, for decades. So I don't think that's in question. I think that we should consider where, really, how are we going to simplify the life of somebody trying to use the cloud, expressing a simple desire to connect their applications to each other and to the internet? And that is not going to be giving anybody direct access to the hardware. I think that's been made clear by a number of folks on this panel. But it does beg one interesting question, which is most of what we've done with Neutron and with current cloud technology is based around IPv4. And IPv4 is a dead end. IPv4 is out of addresses. IPv4 is helping us or pushing us into some of the corners that we have defining network virtualization. There's some nascent work in Neutron to look at IPv6. How do we support IPv6? And one of the interesting components with IPv6 is it's possible that we give every instance in a cloud a publicly routable IP address. That really changes some of the notions of what we think network virtualization is there for. We still are trying to connect applications to each other and to the internet and make sure we have the right isolation and privacy around our individual application containers and their network traffic. But it's not clear that doing that in a way that requires building virtual private topologies is the right way or the only way to do that. So those are the things that I wanted to point out, which I think are fundamentally different from what my fellow panelists said, which for the most part, I just agree with their great points made. Thank you. Refute to us, each other. Thanks a lot, Chris. So, do every of you agree with each other? It's no fun if we agree on everything. It sounded a little bit like it, but I think Chris brought some very valid points to the debate. Why don't you move a little bit more into the light? So, you know, if I can, I really wanted to disagree with Chris here on, you know, most of it, the things he said, I mean, I do respect his opinion, but I really believe there's nothing fundamentally wrong with the Neutron API. I think often people confuse the Neutron API with some Neutron implementations. So, you know, I don't think the Neutron concept of a network is an L2 domain. I- It's actually defined that way. If you read the documentation. It's a very literal interpretation. I mean, the API is- It says layer two broadcast domain. It is just- The concept of an L2 layer two broadcast domain is not, means different things to different people. So, if you talk to, you know, traditional bridging guy, an L2 broadcast domain means an IEEE bridge. If you talk to somebody that wants to pass IP packets, an L2 broadcast domain means that when you broadcast to 255, 255, 255, 255 packets end up on all those endpoints. So, for instance, the implementation I work on does the later without doing the former. You know, really some of these things are given to interpretation. Second point regarding the VPC APIs, the Amazon VPC APIs. I mean, a lot of the people we're working with really want that as sort of an administrative building block. So, you know, the fact that you could have public IPs to every VM does not solve the issue of segmentation for administrative purposes, right? So, I mean, the thing with networking in a data center is it's a form of access control. And you could say, well, and I've heard some people say, well, let's get rid of networking as access control. What we'll do, the only thing we'll do is only have SSL sessions with full authentication and all my VMs will only speak SSL and bring access control to the key management. It's interiorly possible rather in practical because that would require a unique RPC infrastructure across all the infrastructure, everything you use.  For the rest of us, I don't think that's a strategy to replace access control with SSL. To be clear, I wasn't suggesting that. Actually, not at all. In fact, I was suggesting that access control is fundamental to isolation of tenant networks. So it's not about SSL-based authentication to the application, but it's about leveraging the existing network infrastructure that we have to build isolation. And that could be what we've been doing. We started with VLANs and we know that those don't scale and that's part of the layer two issue that I was alluding to that I think you mentioned as well. We've moved to layer two over layer three. So that's the overlay. It's not clear that we need layer two over layer three. I think we could do layer three over layer three and that's really what I was suggesting. And I actually don't think we have the same disagreement. I think we probably agree. I mean, I think L-tree over L-tree is, it's what we've started doing from the get-go and it's actually how you implement the semantics that people need. Nobody really cares about Ethernet headers on a cloud. Yeah, that was part of my point. The other part is with IPv6, it's not obvious that we need that L-tree over L-tree. It's certainly possible that we could do L-tree. In theory. L-tree, just direct connect to L-tree. In theory, IPv6 has mechanisms for mobility but do they actually work well enough? You know, I don't, I think we, I think it's a valid thing that we should push on and explore. Yeah, it's time to find out. No, it's true. I mean, definitely just a meta point here is that we have been designing things based on limitations of technologies and we should think a bit more outside the box. Me, even Amazon's original EC2, right? The reason that the whole model is with NAT everywhere is because of technological limitations. So I think they wouldn't have done that, you know, later on. But I think, you know, Neutron provides really powerful and important virtualization with very little abstraction and it's making the interface more abstract for a certain set of users who don't know anything about network protocols and they're not network engineers. They know that these 400 workloads have to talk to those 200 workloads and call me when you're done. Sure. That's the API. There are orthogonal concerns and we can probably have a much longer discussion about each of these. But just to build on for a second what Dave was saying, that at least a number of implementations I've seen around Neutron do literally take the Neutron network as an L2 broadcast domain. And maybe you're saying like maybe they shouldn't interpret it literally but they do and that leads to all kinds of constraints around the underlying implementation because the definition, if you actually, you know, if you read formally what it says, it says it's a broadcast domain. So they have to honor that semantic even though it's the wrong semantic, a lot of people don't need it and it limits a lot of underlying behavior. If it hurts, stop doing it. Thank you. So there is at least three people that have questions so I don't know which one wants to start. Okay, you're starting. So this is for Pedro and Mike. As performance becomes more of an issue, how does Juniper open-contrail or an OVS, a Cisco ACI approach, deal with VxLan offload on adapters? Yeah, so well, from an ACI perspective, so we think VxLan offload is essential, right? Doing the VxLan in-capping software really doesn't make sense. I don't know if you were here for the previous talk, they had some nice performance slides, you could probably run them yourself and you see that you're getting like one or two gig out of the server and out of your 10 gig, Nick. It doesn't make much sense. The approach we have with ACI today is largely, essentially leveraging the capabilities in the switch to be able to do offload and essentially using VxLan gateway functions in the switch to essentially have full hardware acceleration, full on-rate performance with no overhead. You know, now what we're seeing now- When you say the switch, are you talking about the physical switch or are you talking on offload? No, I'm talking about in the top of rack, in the top of rack layer being able to do- Right, but I still have then the overhead and the OVS switch. It depends what you're using for your driver. So if you're asking specifically about, you know, in the ACI world, there's a lot of ways of approaching this problem. One of the ways we do it is you can actually use locally significant VLANs, which don't have any overhead out of the V- from the V-switch up to the top of rack and then do the VxLan at that point. If that's, you know, if that's what's required, there are a set of NICs coming on board, we'll be using them with ACI, you know, there's architectures that'll support them, you know, other architectures that will support them, where you're using VxLan offload in the NIC, it's also a viable approach. You need to do VxLan offload in one of these places. It's crazy to do it directly. And ACI will support the VxLan? ACI will support it, so right, yes, ACI, so right now, as I said, our solution, we essentially use, you know, we will support VxLan out of, you know, out of the V-switch and software, if that's what people want to use, we don't recommend it. We think you should use, you know, VLANs up to the top of rack, but we'll do the encapsulation in hardware. We're also, again, you know, looking at doing it in the NICs, as those NICs become more prevalent in the servers that people have. Pedro, do you want to add to this? So, I mean, I have a different view. So, at the moment, the performance we measure with two active 10 gig is about 16 gig in iPerf throughput, which if you look at encapsulation overhead, it's just about theoretical maximum. The PPS numbers, though, are not as good as the iPerf throughput, and that's actually more because, typically, we measure with KVM, and there's a performance impact of KVM. Linux Preach is also not very good for PPS numbers. So, what we're actually looking at is looking at using technologies like Intel's DPDK to try to go look into PPS performance, because I think throughput is okay, and improve PPS over what Linux Preach gives you. Which I think is also, this is excellent work too, and actually, there's folks on my team that are also working on the DPDK project. I think that's a very valid approach. Tying DPDK properly into offload techniques on the NIC is still work that needs to be done in the community, so that's actually an important gap still to fill, though. Wow, we've got Jennifer and Cisco agreeing, once more. Yes, this is Michael Delzer, American Airlines. For me, I do believe we need to start focusing more on IPv6, so I agree with the gentleman from Red Hat. I do see that there's a fundamental problem for us as enterprises looking at clouds, and one of the things I think you guys are not addressing that I was hoping you had addressed is enterprises need more than just my data center or a subset of just the open stack in an SDN. I want my entire corporation all over the world to be in an SDN type of an environment. My cost of doing labor today with a traditional network design is killing us, or killing us financially. My intent is to try to have one kind of concept that works and my ability to create micro instances of compute power anywhere in the world where I need that power is what I'm trying to get to, and the idea that I'm having to create just one cloud and that one cloud is its own entity and doesn't really have any relationship to these other clouds is kind of a problem for me, and also the concept of I'm going from a front-end web server to a database server as a siloed end-to-end path doesn't not work for me either. I need a design for microservices. I've already spent 14 years building an environment of SOA services where I have tons of east-west traffic, so the concept of having a linear flow does not work for me. So I need the focus of how I get this power to go forward where we can do a mesh where things can talk to things, where I have the ability at a PAS layer to provide that really abstract layer where people don't have to worry about networking, where the COTS applications with a lot of enterprises have to deal with, they're probably going to be the people who live on open stack and they're not using the PAS layer, they then need a more customized ability to say I have these kind of requirements. For companies like American Airlines, we need multicast. For people who deal with radar data, they need to be able to take that one source of radar data and multicast it out to anyone who wants to consume that radar data. So there is needs for us to have applications being able to consume things in a multicast kind of format in a microservices type of environment. You realize you're asking for a lot. I don't know if I have a great insightful response, other than I did intend to mention and I forgot one of the things that we're also not doing well is connecting applications between clouds. So that might touch directly on your point. It has come up several times over the last couple of days that at the very least what people want is an Amazon VPC peering type of model between clouds, maybe within the same administrative domain with federated IAM, but at least that doesn't seem so hard. I think we can do this. So part of it is the open stack layer has to evolve to be able to present this sort of image of one ginormous cloud that's the whole world and then the SDN layer beneath it also needs to grow and that and a lot of the work in the SDN space is young enough that we just sort of haven't gotten there yet or walking before we run kind of a thing but I think sort of the federation to create big things out of lots of little things is kind of the next big problem. So I mean one of the advantages in our case we built overlays out of technology that's 10, 15 years old which is called BGPL3 VPN. The cool thing about it is you can talk to any carrier and they'll give you an L3 VPN across the world and you can peer between your cluster and your L3 VPN network or peer between multiple clusters. So I do believe that dynamic routing is really important for open stack neutron clusters. Traffic does not necessarily begin and end in the cluster and I believe dynamic routing is required. I think there's some work starting in dynamic routing into Neutron. So yeah, the community will get there. It just takes some time. So the last comment I'll add. I think people touched on the inner data centers. There's a clear need there and there's work going on. The piece I wanted to address was the second part of your point that we're talking about a lot of kind of new cloud architectures where you can blow up a lot of the ways you've been doing work and we can give you potentially IPv6 addresses, your public IP addresses on every machine. There's a lot of new paradigms that might be very interesting that you might be able to move to eventually but it's not what you have today and it's not even potentially what you want out of the gate. So we also have to look at how do we serve the traditional enterprise versus the I'm building a new cloud from scratch and I can start everything from a clean slate. We can't leave some of those designs behind. I think most of the way you'll end up getting served there, best is by working with different commercial vendors. So this is where the work we do on ACI, for example. We have very strong multicast support inside our ACI fabric because we're tying things together at the physical switching layer. So we can solve those kind of feature requests really well. If you wanted to build, but those kind of things come from an enterprise environment, you may not be present, you know, new age cloud environment, you may say no multicast. But I think the way you're going to find those kind of requests best served is you have to go vendor to vendor and see the different offerings that exist and how they can check different boxes. My cast for multicast across the world. That was the part. I don't think ACI is doing that yet. Not quite. That's fair. Hey guys, I'm Karthik with Red Hat. So coming back to some of the architectural comments that Tris had made and Pedro and a few of you had made. So I think there have been some fundamental discussions and debate within the internet community, going back 25 years, 30 years, on the notion of the use of an IP address, both as an identifier and as a locator, a routing locator essentially. And there have been numerous approaches over the last 25, 30 years. Very smart people have proposed ways where you can decouple the two. And most of those are kind of fallen by the wayside. There have been approaches in IPv6 in the early days around multi-homing, renumbering. And I think it would be worthwhile, even if we don't necessarily change the neutron abstraction as presented to the end instance. But at least how we look at the treatment of the underlying neutron abstraction below the surface in such a way that we can decouple that identification purpose versus the routing purpose. And I think some of the approaches that have been proposed over the last 20, 25 years might be worth revisiting to help with things like mobility, to help with some of the other things that we're trying to do in a more artificial way right now, just with an open stack. So I would really invite us to have some of those discussions offline and look at things that have been proposed over the last 20, 25 years. And I think it's a great opportunity going back 20, 25 years. I know there are a bunch of folks within the IITF community going back to the early 90s. So just a quick comment there. I think we all took the IP encapsulation approach or MPLS encapsulation approach precisely to make a location and identification separation to separate those concerns. But it's application transparent. That's nice. I'm not sure about the vast number of approaches that you mentioned, but I think some of them do require application changes and that's not going to fly. Not all of them. By the way, Karthik, we're actually looking at using Lisp to do the cloud to cloud connection so that we can build mapping databases that span. Thanks. You're next. Hi, Anthony from Comcast. I have some questions about how you guys plan dealing with external L2 networks that aren't exactly part of the cloud's specific domain, but may actually be connected to the same L2 segments. A couple of specific use cases. So being one of the guys that did some of the IPv6 work in Neutron, I'm actually interested in forming an L2 adjacency with a router to do MTISIS and potentially using some devices that aren't exactly L2 intelligent to use a L3 gateway that's actually a service VM inside the cloud itself. I can speak for my product. We do EVPN interoperability with existing physical devices.  Yeah, I mean like provider networks turning off the L3 agent kind of thing. Yep. More than just that, you turn off everything, so it becomes just a ether network. But the provider network, that's the only primitive we have in Neutron to do what you're asking. So much for the notion that L2 broadcast segments aren't used, right? There's always exceptions, right? Exactly. I wish they would just go away. There's always exceptions. I mean, but that's not probably your number one design point. I think you don't need to design for making L2 your primary consumption. Can I just say one thing? I think one reason why we don't make more progress is because we're addressing several use cases that are all different. If we're all building large clouds we would do everything with L3. If we're dealing with enterprise we would do things in a different way. That's right. I 100% agree. The environments in many cases are different enough that you layer on a lot of complexity when you look at the enterprise environment that you can strip away in the cloud environment. I think we are over time. Unfortunately, we could have continued this discussion easily in our drinks. Thank you very much for your help and for the room as well. Very interesting questions. See you soon.


So today guys we have details on the next DLC for Borderlands and it looks pretty amazing.  Okay so if you didn't catch the latest Gearbox N2K stream I have you covered, we have many details on events to come and much much more so let's check it out. Okay so on March 26th we are getting the second DLC campaign add-on, this state will be a surprisingly dangerous engagement party. The DLC is called Guns, Love and Tentacles, The Marriage of Wainwright and Hammerlock. Now looking at this image on screen now with Hammerlock and Wainwright we can see this heart shape with these tentacles and so forth. Now you guys may remember me speculating quite a while back with the Broken Hearts Day event, with a few of the items we saw including the logo showing a heart with tentacles, but the clues have been there since the start people, sneaky sneaky Gearbox. So sir Alistair Hammerlock and Wainwright Jacobs are finally tying the nuts and they want us to celebrate with them on a new planet and destination to the game. A planet which is called Xylaga something that says Xylaga so I'm not exactly certain that's a hard word to say for me. Newbie Englishman, now this planet is described as a frozen plane in which its wastes have thinned out all but the vicious wildlife and the e-return of Curse Haven, where we all agreed to meet for the celebration of Hammerlock and Wainwright. This is a place located beneath the colossal carcass of a fallen vault monster. And guys, if the ice swirls biting cold and vicious wildlife weren't unwelcoming enough, a band of creepy oculus who worshipped the vault monster's carcass are how bent on breaking up the betrothed partners. You will have to fight back the forces of unworldly evil if Wainwright and Hammerlock are to have their happily ever Luckily, the most badass wedding planner in the galaxy has your back. Gage, a fugitive vault hunter and returning fan favourite character who's inseparable from her robot BFF death trap. So it's absolutely amazing that we will see Gage back people. I know she was a fan favourite to many many people so yes I can confirm she will be back. But people that's not all. Reunite with old friends and team up with some eerie new characters to help true love triumph over edge rich horror. Confront bloodthirsty enemies and formidable bosses including alien beasts, hideously mutated oculists and a clan of cold blooded bandits or enemies you can see within that grind now. Delve into the disturbingly and occasionally death backstories of cursed tanfolk in a host of spine tingling side missions. Arm your vault hunters from the deadliest gear in the galaxy including a trove of additional legendaries and some supremely powerful class mods. Flort your sense of style with a slew of new cosmetic items from vault hunter heads and skins to interactive room decorations and all new vehicle skins. So yeah people how amazing just this DLC sound. And the guns, love and tentacles will be available on March 26th which is a part of the season pass for Borderlands 3. I personally cannot wait for this people it's going to be incredible. Again March 26th mark your calendars but those days off work because you know what we are doing. So yeah that's all we know so far on the new DLC which comes out next month but obviously if any more news drops surrounding this I will have you covered right here on my channel. Now lastly before we end people Borderlands 3 is going to steam here are the details we know. On March 13th Borderlands 3 will launch Unsteam with PC crossplay between Steam and the Epic Games Launcher thanks to some new shift matchmaking functionality. You'll be able to add friends from your friends list join co-op parties and play together online regardless of which PC platform each player is using. Gearbox plans to roll out additional PC crossplay features such as the ability to mail weapons across platforms. This will happen in the weeks that follow the games launch on Steam. So for those PC players out there watching this this info is for you obviously. But that is that guys for another Borderlands 3 news video I do hope you enjoyed it if you did leaving a like really helps out. If you're new around here want to see more Borderlands be sure to subscribe and if you never want to miss a video upload you can turn notifications on by hitting that bell button. But guys thanks as always for stopping by hopefully you enjoyed the video and hopefully I will see you on that next one