Eric Raul, if you could join us in the attendee chat, that'd be awesome. And thank you for joining us. Really appreciate it. All right, Christy, let me know. All right, hitting the bumper. Hello, everyone. Welcome again to another OpenShift Commons. And today we have a very popular topic. And I know we all, we all love networking. And this continues our series of what's new in OpenShift 4.8. And we have lots of great new networking enhancements. And we have Mark Curry here to give us the update and please get your questions ready. Put them in Q&A, put them in chat. If you're joining us through OpenShift TV, we have people watching those channels. So put your questions there as well. And let's go ahead and get started. Mark Curry is one of our consulting product managers. And if you know networking and OpenShift, you've already heard his name. So he doesn't need much of an introduction, but Mark, so excited. Thank you for joining us today. And you want to go ahead and dive right in. Okay, thank you. Let me go ahead and share my screen. All right, hopefully, if that has not already popped up in your screens, it will very shortly. So first, thank you, Karina. And also, thank you, everybody, for your time today to talk a little bit about some of the new enhancements and the features that you're going to find in our forthcoming OpenShift 4.8 release. So let's first take a quick, big picture look at what I'll cover today. So some of the things captured here, you can see that IPv6. We're going to talk a little bit about new router configuration settings for HA Proxy. Some of our observability effort in the form of NetFlow support. Some upgrades to some of our core products, that is core DNS and HA Proxy. And some of the advantages that are brought forth from those upgrades. We're going to talk a little bit about the oven migration tooling and some enhancement to that. Talk a little bit about SRIOV NIC supports and one big thing that's not technically 4.8, it's 4.9, but still worthwhile knowing for those of you that do SRIOV networking. And then we'll talk a little bit about network policy, audit logging in events, and network policy of Mac VLAN interfaces. A couple of things that are not in this topic list, which I threw in fairly last minute, I'm going to talk a little bit about a couple of new D&I plugins that we support as well. So there are many networking developments that we did during the 4.8 timeframe, but really these are probably the ones that are most requested or are most visible changes to our customers. And so that's where we'll focus the topics of the presentation. Everything is fair game for questions. So first up, let's go ahead and talk about one of the bigger things to have happened to Kubernetes networking recently, and that is IPv6 single and dual stack enablement. So we've supported IPv6 for secondary interface, data plane interfaces for a while now. But as a 4.8, which is built on Cube 121, where IPv6 dual stack first went data and the API finally stabilized, OpenShift now fully supports IPv6 end to end on the control plane of a networking bare metal deployment. So we'll add other platforms as we can that are IPv6 capable in the future. So backing up a little bit to make sure we're all on the same page, single stack of course means that the pod interface has either one of a v4 or v6 address on it. So you choose one of those and all of OpenShift networking is going to be 100% aligned to that choice. Dual stack on the other hand means that the pod interface has both v4 and v6 addresses assigned to it, as depicted in the graphic that's on this slide. So what that means is that the cluster can communicate with any internal or external endpoints that are using one of v4 or v6. The latter configuration v6 represents the vast majority of our customer use cases, mostly because single stack is, as I said, 100% single stack, 100% v4 100% or 100% v6. If you need to connect even one host of the other IP family, then you need dual stack. And we've had several customers that were affected by that scenario. And so most times, unless there's a very strong need for single stack, most customers will choose dual stack. So how do you enable v6? It's actually very simple. At install time, you do that typical pause of the install to modify the install config gamma file. You specify IPv6 subnets in addition to the IPv4 subnets. And then just continue with your install. If you're trying to do this post install, you can still do it. It's still pretty straightforward. You simply edit the network config resources to add secondary, you're going to need machine network, cluster network, and service network values. And then those will get rolled out by the cluster operator correctly. So it's pretty straightforward to do. The common question I get is, what about other platforms? You know, as I mentioned, urban bare metal, but, you know, there are, we're definitely working on some other on-prem platforms that support IPv6, for example, open stack. But there are no near-term plans to port v6 on most, if not all, the cloud providers, at least to end the way. And so because of some of those current issues on those various platforms, we're blocked. So, for example, some of them only offer their APIs and their DNS over v4. So you can't do dual stack. At least one of them has a load balancer that accepts v6, but translates it to v4 internally. So it's not really doing v6 and so forth. So there's other issues. So until and if some of those issues are ever resolved, this will likely be the status quo for a while, platforms we support today. Next thing I want to talk about, and there's a lot of content on this one slide. So there are many new features and enhancements to OpenShift Networking 4.0. As I mentioned, I'm going to cover some of them here. And then there's even more I'll cover in the next slide, a very similar slide, as a sort of representative cross section roughly divided into ingress, egress, enhancements, and general networking enhancements. So this first slide, kind of roughly across ingress, egress, enhancements, one big thing is that in 4.8, we upgraded to HA proxy 2.2 long-term support inversion. Our next update is targeting 4.10, where we'll go to HA proxy 2.4. So we get all the new features that are detailed in a link that I believe is down the student note section of this slide, which will be shared with you, including some of the major ones listed here in this bulleted section. So there's going to be increased performance. There's all kinds of enhancements for security, hardening, and options. There are health checks that are added. There's new and improved observability into the traffic and the workloads that are happening so that you can do better debugging. Another big one, you can do syslog over TCP. There's also new SSL TLS capabilities. There's a stronger encryption 2048 bit, dynamic SSL storage, and so on. If you want to see the full list rather than me talking about them highlighting them here, definitely go and check out the HA proxy 2.2 release link. Along with the HA proxy upgrade, we added several new supported customizations for it. So one of the first ones that we did was router use proxy protocol, which basically allows the source IP address to pass through a load balancer if the load balancer supports the protocol. One example of that is Amazon's ELB. So we have a number of customers that have asked for that functionality. Another one, router back-end process endpoint. So this is critical to shuffling endpoints for proper distribution of the requests when you're running multiple routers that have a load balancer in front, like, for example, if you have an F5. So this will help balance out that load internally on those endpoints a little bit better. A couple more options, tune max rewrite, tune buff size. So some customers have the use case of very large header data. And I'm talking about on the order of about 48K, somewhere in that neighborhood, 48K, 64K, which is very large and maybe was a bit unexpected on our part. But if HA proxies buffer for that header data is not large enough, then that data gets dropped. So we now support the configurability of these parameters to allow for that. We don't limit the value that can be used. However, keep in mind the larger the cluster, the more memory it's going to consume as that configured value goes up. Another one, customizable number of router threads, also known as NB thread parameter. So since 4.1, we have a, we've supported the NB threads parameter that defines the number of threads used by HA proxy. But what we did is we did an engineering analysis and we determined a best practices value for that to be 4 threads, for many, but not all workloads. So what we did right out of the gate was we assigned a fixed value of 4 threads to NB threads. However, customers with large cluster nodes have asked us to make that configurable. So that they can accommodate some larger host nodes that they happen to be using in their deployments. So what we've done is we've made this now customizable in 4.8. Searching gears a little bit, IP failover, keep alive D support. So the keep alive D image, it's interesting. It's been there in the product for a long while. We dropped documentation for it, but we never dropped the keep alive image. And so it kind of was this, a bit of a hidden secret within the product. Customers still wanted to use it and they wanted our formal support of that. So of course, what we did is an open shift was we formalize support for the use of that image to provide high availability in open shift. And as part of that support, we of course needed to introduce brand new documentation and a procedure for implementing it on cluster nodes. So, or not a brand new, but rather a current procedure for implementing. So that's what we've done in 4.8. Other thing gateway API. So in open shift 4.8, we will present a developers preview of gateway API. So this was formally known by several names. This was known as ingress v2 in the very beginning. It changed names to service API, but now it's changed to gateway API. Hopefully that sticks. I think it's the most appropriate name for it. But gateway API represents a unifying technology for ingress. And so we're targeting integration of it with the likes of another project which may be an envoy friendly like contour as the primary ingress controller for traffic alongside HA proxy. So that would represent an enhanced integration with, for example, open shift service mesh with its envoy project involvement. Of course, you know, there's going to have to be a great deal of performance and scale testing to enterprise hardening to make sure that that is going to serve our customers in a positive way, but that is underway currently. Another thing global access option for GCBs ingress internal load balancer. So without this particular option, traffic originating between projects in a shared VPC network must be in the same region as the load balancer being used. So this facilitates communication cross region for shared VPC deployments. And finally, another one, egress IP load balancing enhancement. This is for our current open shift SDN customers to spread traffic more evenly across cluster nodes. So what this does is this removes that single node choke point of egress IP where the IP address is assigned to one node in the cluster and therefore all traffic must go out that node to be assigned that egress IP. So you remove that choke point and then you can spread the load across multiple nodes and multiple IP assignments across cluster posts. A similar OVN enhancement is forthcoming in the... All right, so let's click gears a bit and talk maybe just a little bit more about just some general networking enhancements. Again, this is a small sliver of a much larger effort that's going on, but these are some often requested or asked about things. So a big one is network observability. We have a rather large effort underway across all of open shift or to increase observability of all the things. Specifically for networking, networking complexities make cluster administration really difficult even for experienced administrators. So the goal of networking observability is to improve the quality and visibility of open shift networking metrics, some of the important networking correlations that are inside the UI or that can be drawn for you for an improved operational experience. So one of the big things we've done specifically in 4.8 towards this goal is to provide network flows, tracking and monitoring for network analytics. And so this will help us with a supported way to monitor traffic into and out of the cluster. This specific enablement is a first step in preparation for adding a NetFlow, SFlow, IP fix collector to oven Kubernetes in a follow-on release. So this is going to be really helpful for troubleshooting performance issues, capacity planning, security audits and similar, and look for much more from our observability efforts in follow-on releases. Also in 4.8, we've added some key SRIOV capable NIC hardware support for our customers. And so a lot of these requests came from customers, partner organizations. And represented here is sort of the short list of what we did specifically in 4.8. So we've added the Mellanox 6.5 family, we've added Intel Columbiaville family, HPE Ethernet adapter, specific adapter. And I know this presentation is really about what's new in specifically in 4.8, but I do have to mention that in our follow-on release, 4.9, we've integrated with RHEL's fast data path team and their test harnesses. So at 4.9 onward, any NIC that RHEL supports for SRIOV, so will open shift. So this list that presented to you of three different adapters, hopefully I won't have to keep presenting adapters going forward. In 4.9, we do have some specific ones we added in parallel to this harness integration with RHEL, fast data path. But hopefully in the future, this will be just something that we looked up in our documentation to see what the RHEL supports. Look for more on that in a future release. We've also added network policy support to Mac VLAN interfaces for our customers. So when you're using Mac VLAN, kind of a quick background on this, a virtual network interface gets created and it then gets exposed to the local switch with a new MAC address. This interface is in turn bound to the container. The advantage is that from a networking perspective, it looks like the host has two network interfaces with one of those being assigned directly and exclusively to the container. So the upside to that is since the pod is exposed externally and directly bypassing the SDN, users can realize greater performance and they can also use more protocols. Downside is that there is no routing layer now in front of it to protect it. So you could use something like IP tables or another firewalling technique inside the container, but you'd have to do that for every application which can be operationally difficult. So we enabled network policy from Mac VLAN interfaces to protect that traffic. And we're doing it in a globally configurable and accountable way. So as network policy becomes more global, more cluster-wide configurability options are added, as well as multi-cluster, this will be just part of the larger plan and vision for that. This next one, I'm going to mention this one quickly here because I'm going to talk about it in more depth in the following slide, but we've enhanced the OpenShift SDN to oven Kubernetes CNI migration that we support and includes support for UPI deployments as well, which pretty much completes our picture. So I'll come back to that in a moment. For security and compliance reasons, our customers have also asked us to provide and enable audit logging of network policy events. So what I mean by that are, for example, when a packet is either accepted or denied, that largely, not largely, it went entirely missing from any sort of logging or auditing events. So what we've done is we've added that back into the product. So this information is presented to the built-in logging stack and custom Kibana dashboards and is useful to support our customers in their compliance security policies where activity on the far wall needs to be inspected at runtime to monitor suspicious behaviors, like basically act in the role of an IDS for intrusion detection support. Or even for post-mortem analysis. Lastly, there's been a couple of product updates within OpenShift Networking Core DNS. We updated it to version 1.8, and this is going to include a number of feature enhancements and bug fixes. With 1.8, we also provide the ability to control OpenShift DNS pod placements. So this is for those customers with extreme workloads that need to be able to control where exactly DNS runs, either so that it's potentially locally available to where it's needed for optimal functionality or maybe to isolate it so that workloads aren't, or DNS itself isn't bothered by the workloads or vice versa. So this is a big part of Core DNS 1.8, some of the functionality we've had. All right, I mentioned that I was going to come back and talk about OVN migration tooling. So as I previously mentioned, other migration tool support is now supported on all platforms that we support, and this includes those platforms that are both UPI and IPI installation modes. UPI, I don't want to presume too much here, user provisioned infrastructure and installer provisioned infrastructure. So whichever mode of installation you chose, no matter the platform, we support the OVN migration tool on that platform. So OVN or OVEN is supported as of OpenShift 4.6, but it's not the default out-of-the-box networking. So customers that want to take advantage of the latest feature enablements and enhancements that are really being done in OVN, they might consider switching from the OpenShift SDN networking plug-in to the OVEN Kubernetes plug-in. We want to make that as easy as possible for them. The easy option to get to OVN is obviously just do a green field install of a new OpenShift cluster and just choose OVN at that point. However, for customers where that is not an option and still want to use OVN and just convert their existing clusters to it, we want to make that option as painless as possible, but understand that swapping networking on a cluster is a non-trivial process. And there is likely going to be some amount of service disruption. We try to minimize that wherever possible. And so typically what people will do to mitigate that also is to schedule this during the maintenance period. So the graphic on the right outlines a little more detail of the procedure for the migration. And this is also going to be in the 4.8 documentation. And also recognize that this is for going from OpenShift SDN to OVEN Kubernetes, but there is a similar process, different but similar process, to roll back if you ever needed to do so. Specifically, this is migration, and I should point out, this is not migration from just any SDN plug-in, a CNI plug-in to another. This is specifically supported and fully tested and vetted between OpenShift SDN and OVEN Kubernetes. So this particular migration is logically split into two phases. One where the machine configure operator prepares the nodes for the new networking. So in that first phase, this involves a series of, excuse me, a serial reboot of all the nodes in the clusters. So larger clusters will take longer to migrate. This is the current way that this is done. We're looking for a way to optimize that so that this is not done serially by the machine configure operator. And so look for that in a forthcoming release. In the second phase, the cluster network operator, I'm sorry, the backup. So after the machine configure operator prepares the nodes, the second phase, the cluster network operator, is going to deploy the new control plane to all the nodes. Once again, once you change that, do the actual flipping of the configuration over to officially to OVEN. Once again, you're going to have to reboot all the nodes, but this time they can be rebooted all at once. And in fact, that's desirable. So they come up with a cohesive understanding of the new networking. The timing issue just with one of the nodes and the networking didn't quite come up properly. Just reboot it again. It'll come up fine. And so that's the process. So we've tried to make this as painless as possible. Keep in mind that it is, like I said before, you're basically doing open heart surgery on yourself and you're doing migration. So I'll plan accordingly. And the last slide I want to include in here was to talk a little bit about some new plugins. So this is not really tied to an actual development effort in the 4.8 release, but these did happen. These certifications did happen during the development cycles of 4.8. So I think it's worthwhile to bring up here and discuss. So the two new Kubernetes CNI plugins for OpenShift that are certified and supported are Isovalence, Stilium as an option, as well as VMware's Entrea plugin. So Red Hat does not provide support directly for these plugins, understand. So that's never been the case. The only ones that we provide direct support for are the ones is obviously OpenShift SDN, our current default, Oven, our next generation networking, and on the far right, Career Kubernetes for those people that were looking for the equivalent of a pass-through of networking down to the underlying OpenStack Neutron networking layer. So we don't provide support directly for any of the other ones, but we partner with the organizations responsible for those CNI plugins to provide support. And because of that, I just recognize that different third-party plugins have different levels of testing and support for our other layered products, such as OpenShift virtualization and OpenShift service mesh. So consult your account team for more information if you have any questions about that. But we welcome on board these new plugins and hopefully these will be useful to our customers going forward. And so that brings us to the end of what's new at a high level of networking in OpenShift 4.8. So I will open the board to any questions that people might have. Thank you, Mark. You have this timing down. Perfect. Okay. We do have a few questions. First of all, can you put it or maybe stop presenting? Sure. I love the thank you slide, but... Yeah. I mean, I might have you go back to a couple of slides if they make sense. So first question. Do OpenShift routes support non-HTTP protocols like RTMP? I tried to set up my own restream.io-esque service and had some issues with it. So the protocols that are supported with routes is HTTP, HTTPS, DLS, SNI. If you have something other than that, we do have several other mechanisms to get that traffic in to include node port, to include exposing external IPs. Those are probably the two most popular ways. One of the things that I talked about during this presentation, you may recall I talked about gateway API and how it is that we're trying to unify ingress. I fully recognize that I'm telling you to do something in different ways simply because you're using a different protocol. And so one of the goals that I have for a longer-term vision, actually not even that long, but maybe midterm vision for OpenShift networking is to unify the way we do ingress so that you shouldn't have to care what protocol you're using.  It really shouldn't matter what you're doing. You should be able to configure your ingress the same way. And this is our goal. That's not the reality today. It can do it. And our documentation should cover that. As a starting point, as I mentioned, it's not about exposing node ports and external IDs with services. But in the future, we want to bring that together.   Maybe go directly via some SRIOV mechanism to the pod. So we want to provide some of those high-performance pathways and networking and completely get out of your way all together. You, of course, would need to handle that traffic yourself and your application. But we want to be able to provide that option. Awesome. Thank you. And if that give a follow-up, please put it in the Q&A or any new questions because, you know, we have Mark here. And he's so in demand. It's not often that we get time with Mark. All right. Is it possible to turn off the network observability features? I assume it will introduce additional CPU costs to the cluster and some people may be sensitive to it. What do you think? Great question. Yes. So we generally have, we've gone through several incantations of, and this is not specifically limited to networking observability, but we've gone through several incantations of how we would deploy this. And in the beginning, we had several different layers of robustity that could be configured. Ultimately, it turns out that our users are really asking for one of three settings on some of this observability, and in particular, any observability elements that are going to add overhead, like additional logging to the cluster. So the three settings are just turn it off altogether. I don't need that information. And you can do this on a sort of a, we're setting this up to be more of a per observability function kind of thing. So you don't have to turn it off globally for all things, but you can target individual things that you'd like to turn off. The other two settings are give me a reasonable, and reasonable of course is very subjective, but give me a reasonable amount of robustity that for some of the things that customers generally ask for, or that we and our own debugging and triaging of customer issues that we encounter and find very useful information. So turn that on. And then the third option is just turn it on full blast. So you might, you know, this is definitely going to incur the most overhead. So why on earth would anybody want this? Generally speaking, this is for, you know, real, really, really in-depth debugging sessions, but probably more likely for those customers who have regulatory compliance that they have to meet that says you need to capture everything that happens, so that if, for example, there was ever a break in, we can go back and do some retrospection to understand exactly what happened, how they were potentially able to break in, what they did once they were in, and so forth. So I expect that one of those three settings will be applicable to most, if not all of our users. I'm happy to entertain additional options, but that's the target for these kinds of things that might incur overhead. Awesome, thank you. All right, if not using a load balancer in front of the OpenShift cluster, is it possible in a supported fashion to load balance the API ingress, which runs in keep live D on the master nodes? If so, are there any documented links? I can go get those. Yes, so great question.  It's really more of a high availability kind of thing. So what our customers do today, and one of the things that we support with our partners is the ability to use something like, say, a commercial product like an F5. You put that in front, you use that to do the load balancing, and we have, if you consult our knowledge-based articles, there is documented procedures for doing that.  But we do provide a procedure for doing that, and if you have F5s, presumably you have support for those, and so work with that provider. In the future, one thing that's really interesting that's coming in 4.9, I know this talk was about 4.8, but I don't think I can get away with not mentioning it. Starting in 4.9, we're going to be fully supporting Metal LB. So if you have a metal, bare metal deployment, Metal LB in 4.9, we're going to be supporting Metal LB Layer 2, and you can use that. Also in 4.10, we're targeting Metal LB with BGP support, and potentially support that. So the BGP support will essentially, mostly the purpose for that is to advertise routes to Kubernetes services using BGP, but I think some form of Metal LB may be useful, and we will provide some more information about how to do that as we get closer to the release of Metal LB. And for that question, he did provide additional context in that it was for VMware and using IPI install, if that helps. Yeah, the specific details about that probably would not be able to answer it's a different way for you than I already have. And it, you know, the advanced, well, rather the custom networking configurations that you can do for VMware installs fully documented in our product documentation, anything you're doing beyond what's specified in there. Like I said before, it essentially would involve the integration with a third party product. And so some of the support for that would fall onto that provider. All right, thanks for that additional information. Another question. Are there any plans to support external load balancers that do full re-encrypt for all inbound routes? My understanding is that currently OAuth does not work with re-encrypt due to MTLS. That is correct. It does not work. And we are discussing that right now. That's a pretty difficult problem to do in a universal way for our customers. But this is on our backlog for feature enhancements. And to date, we are in the design and architecture review of that possibility. All right, please add any more questions into the Q&A or the chat. So Mark, what are some of the common use cases that you're seeing that you're being asked for by customers? Give us an idea of the strategy going forward. Yeah, you know, really top of mind lately has been our customers as not only as the product has matured, but we still have our customers' workloads and how it is they use Kubernetes platform to deliver upon those workloads. That sophistication is growing at a pretty rapid pace. And so the sophistication of the networking, some of the advanced networking features, these are things that three years ago would have been unheard of to ask for or would have raised a few eyebrows that they would have asked for. But with the adding of the telco market to the scene of Kubernetes platforms and all of the requests they have, it turns out that a lot of those advanced networking features that we always thought were pretty limited to telco organizations, turns out those are actually super useful to some of our higher-end enterprise organizations as well. And so I'm seeing an increased adoption for and requests for enhancements to some of those features. So hey, I saw you did this. Can you also support this or make this configurable for us? So look for a lot more of that. That's not going away and that's only expanding. Another really big thing is scale and performance. So again, as our customers' deployments grow and sophistication and size and scale, a really big thing that I'm seeing is people are essentially redlining what the capabilities are of networking. So what we're trying to do is approach that from several angles. The first is better documentation. So one of the things, a very common thing I see is people have completely overwhelmed their ingress and come to find out that they have essentially a choke point in the sense that they're only running one and they're not doing any sort of sharding whatsoever to split up the workload across different nodes in the cluster. So, you know, and when I look at our documentation, I somewhat agree with some of the sentiments I've heard recently, which is it doesn't really fully explain that. It doesn't explain best practices and it doesn't maybe provide really great guidance for how to do that. More importantly, and so we're improving that for sure right now, but more importantly, to a certain extent these customers shouldn't have to worry about that. There should be some sort of built-in intelligence and automation that says it's time to start sharding. Time to break up this workload into multiple shards that we spread out across nodes and have this all done sort of automatically for our customers. So a very recent request for an enhancement that we got really brought that to a head and there are many customers out there that would benefit from this and have now said, yes, please, we would love to see that. So look for more around that in a future releaseable. Those are really the two, probably the two biggest things I get. At least those are the two most top-of-mind things for me. Thanks. Is there anything that you've seen that has surprised you recently? Like you thought it was going a certain direction? I think, I guess a very recent thing is you heard me talk earlier about some of the new CNI plugins we have. One of those was Stilium. So one thing that maybe surprises the wrong word for this, but one thing that I'm paying close attention to here is the adoption and asks around EBDF for OpenShift Networking. There are many customers out there with use cases that would benefit from EBDF being introduced into OpenShift Networking. I am fully aware of that and we are absolutely going down that pathway. In the short term, there's Stilium. I would have expected to see greater adoption. Stilium has been certified now for one, maybe two months. That's probably closer to two months. I would have thought I'd see greater adoption, but I have a feeling customers might be waiting for us to more natively support EBDF in our networking deployments. Please understand that that is forthcoming. When you provide the greater control of EBDF to users, along with that comes some additional questions about how do we support that? If we give you so much more control, you now have the ability to do a lot more things, how do we support that? There's questions around that that still need to be resolved, but this is something for you to look forward to. That sounds like we have a lot to look forward to, at least from the last 45 minutes of talking networking and OpenShift. There's lots of great new features and enhancements.  I know networking is really exciting, but what about you, Mark? I'm very excited about the forthcoming BGP capabilities. We have a number of customers that have been asking for BGP, and in fact, they've chosen a different CNI plug-in specifically and only because it happens to provide BGP capabilities, and our default out-of-the-box networking has not supported that in the past. Look for that starting in 4.10, as I maybe mentioned earlier, but this BGP capability is really going to open up a lot of doors, and it's going to, you know, for some very large customers we have that prefer sort of a flat network in these routable service IP addresses and the advertisement of those routable service IP addresses. They are very, very much looking forward to that, and so I'm pretty excited about the future of that. I'm excited about some of the new avenues we have as I mentioned with EBPF and actually how that EBPF will provide enhanced observability in the product. Some of the integration that we have with some new layered products like our advanced cluster security, ACS, multi-cluster networking being provided and viewed and managed by something along the likes of advanced cluster manager, ACM. I'm also really encouraged by a big effort that we have underway right now to, like I mentioned before, unify Ingress and provide this global API for networking.  So if you wanted to move from one cloud provider to another cloud provider or to bare metal or whichever way you're moving, you change that outer layer that defines the API surface for that provider and underneath that, you know, it sort of quote-unquote unlocks additional API capabilities to configure some of the specifics of that particular cloud provider. Like, for example, if you were on AWS, you would unlock one of three load balancers, ELB, NLB, ALB. If you happen to choose ALB, it would unlock additional configurations and capabilities around the ALB's WAF, the web application firewall functionality and you could configure that. So having this global API to do all of those things is absolutely the direction we're going to for an improved operational and continual management, especially as our customers continue to grow to multi-cluster deployments. Don't get me started, Karina. I'll talk over. I was just thinking, I mean, we'll talk offline about all the other exciting stuff, but mentioning, let's say, the different workloads and we have a question on whether there are plans to segregate traffic on different uplinks based on the types of workloads and make OpenShift workloads aware of that. Yeah, well, that's a really great question. So it is, there is a huge amount of momentum within the OpenShift engineering and business side of things to add increased integration, enhanced integration with post-level networking. So we're an operating system company as well, right? So we have this phenomenal enterprise operating system right at Enterprise Linux, Ralph. And it has an amazing array of capabilities for how it is that you can configure networking, how it is that you want to configure the different nicks in your hosts, how you align interface to those hosts, how you do bonding across those interfaces, all the things. Our customers are asking us to bring that level of configurability that's NREL networking into OpenShift. So today, you install OpenShift and it uses whatever the prime, by default, it uses whatever the primary nick is in the box and it binds the primary gateway interface to that nick. So all your Kubernetes control plane traffic flows in and out of that primary interface. But customers don't necessarily want that. Going back to my earlier conversation about how some of our advanced customers are now asking for advanced capabilities. So now what they're asking for is, well, we don't necessarily want all of our control plane traffic going out that primary interface. We'd like to instead maybe carve it up and send it out specifically to a secondary interface and use that primary interface for maybe data plane traffic or a specific applications traffic. And maybe we want to tie that to a very specific nick or a set of nicks, maybe two nicks that are tied to a couple of top of rack switches and have bonding between them and all the kinds of settings. It's not just interface configurations but host-level networking from the perspective of, you know, some of the telco use cases require, especially with these real-time component requirements, they require things like precision timing protocol. That's a host-level thing. That's in the kernel. So how do we enable that in a meaningful way for OpenShift workload? By the way, we do support today PTP with OpenShift. But that's an example of something where we needed to increase or enhance that integration and involvement and manageability of host-level networking from within OpenShift to service the workloads that people have. So that's a pretty long answer. I hope I answered your question, but maybe the DLDR on that too long to read is we are trying to bring host-level networking into a tighter integration with OpenShift networking so that you can carve up your networking at the host-level any way that you want to do it, just like you can with Realm. You're adding all these new, well, improving on all the enterprise-level features because I think host-level networking and bringing it in is more maturing, right? Networking at OpenShift. And this becomes especially important in bare-metal deployments, right? Because in some cloud providers, I don't have access necessarily to some of the hardware-level capabilities that these instances are running on. But for bare-metal deployments, I presumably have full access or the user has full access or knows somebody who has full access to the host to be able to configure things the way they want. And so providing that for something like bare-metal deployments, OpenStack deployments, this is super important at that level. Sounds like you're really setting it up for the Metal LB Support in 4.9. Absolutely. I was just on a phone call with our bare-metal product manager before this to talk about how we're going to align. Awesome. All right, another question, since we're getting down to the wire, considering networking inside OCP being more mature than before, do we have native network diagnostic tools and route topology? You expose it in some sort of operator dashboard, which gives log analysis, packet drops, detail, formation of flow. What do you think? Yes, good question. So as part of our much larger OpenShift observability project, we have, and specific to networking observability, as I mentioned before, this was really catalyzed by our telco customer base. For telco developers, networking is their business. So whereas some developers, they don't want to have to think about networking, they just want their application to work. But for a lot of telco developers, networking is their business. They need that visibility into exactly what's happening. So some of the things you're going to see that are forthcoming that are in the works is, for example, today with OpenShift Service Mesh, we incorporated the Jaeger project to provide tracing capability of traffic. There's no reason why we can't add that to OpenShift at large. So we're looking to incorporate some packet tracing capabilities. OVS flows today, we have users that want to see exactly what's happening with those flows. So what we have in starting in OpenShift 4.9, the ability, we have a very simple tool that will present these flows to you in a very simplified manner. And I just saw a wireframe targeting 4.10 that will further enhance some of these flows and bring this really nice interface into the actual OpenShift console, present this much deeper inspection of packets and packets that were dropped, traffic, throughput flows that are active and so forth. And so this will help as a starting point provide that kind of information in those metrics that the developers and even the cluster administrators or project administrators require to understand about what's happening with their particular projects.  There's an improved visibility into the network policy, being able to view that better. So for example, today if I said, hey, how do you have network policy configured on your cluster? You can spit out for me some YAML. I've been doing this for a long time and I look at that YAML. My eyes are going to glaze over a little bit because it's not that straightforward to read, but it is readable, but that's not the way security should be answerable. So what we have done is we are making some of the network policy configurations more digestible by the people who need to digest it. So for example, network administrators, security administrators, they're used to talking in the language of ACLs, zone rules. They work on these things on a daily basis on their switches and everything else that they interact with. So why not present network policy in that format? Maybe present it in a graphical format, so that a picture truly is worth a thousand words, where you say, hey, that red line or that line that doesn't exist between these two objects in my project, that's a problem, or maybe that's by design. And you use that to confirm that you established proper network policy to limit who can see what within your project. So all of these things are coming to bear. I would say 4.8 was really where the gears started to turn to produce some of these enhancements to observability, but we're at speed. So look forward to 4.9, 4.10 to see some big jumps in that capability. Awesome. There's not many people who can make networking sound as exciting as you want. All right, we have a final question. Is it possible for CSI driver to mark traffic with IP DSCP values? These DSCP settings will be used on ACI switches for delivering quality of service. Storage traffic shall be prioritized and not dropped if marked with the correct DSCP settings. So when I hear CSI, I immediately think storage, but if you're talking about IP storage traffic, I think what I heard was IP storage traffic and specifically the QoS of that traffic. Today we have some QoS capabilities on SRIOV traffic, but we do not have it in the larger... We don't have it upstream in the larger Kubernetes ecosystem. So one of the things that we have started is to upstream some QoS end-to-end capabilities within Kubernetes to allow for some of these kinds of rate limiting that you would expect from QoS. And it's not just storage IP traffic. There's a lot of customers that want to be able to put caps on basically any kind of labeled traffic from their applications. Maybe they want to rate limits backups. Maybe they want to rate limit a specific application. Or maybe another one I've had a lot lately, the ability to rate limit incoming traffic for two reasons, to mitigate distributed denial of service attacks, as well as some administrators have asked me for the ability to rate limit to limit the streaming of images from, say, a QA registry into new projects, because technically somebody could denial of service from within. So they've asked for these kinds of capabilities. There's different ways to attack this. Maybe we build it out and create self for that kind of use case. But still, there's many other use cases in the larger team of things that could benefit from end-to-end QoS within Kubernetes. And of course, that's where we always work. We do upstream first. And then we would enterprise hard and then bring it down into open shift. And I love that we are ending on you saying we do upstream first. So if you're wondering what's happening with networking as well, just go look upstream, right? That's correct. That's where all the action is. All right. So go get involved if you aren't, and you really want a feature pushed. That's where you can do it. And thank you, Mark, for giving us this deep dive into open shift networking and what's happening in 4.8 and some futures. Is there anything you would like to leave us on as we're closing out? We are customer-centric in our roadmap. So you heard something today that you would like to see done differently, or if you didn't hear something today that you wish were in Kubernetes or open shift, please reach out. We are customer-driven. And your requests of today could be product realities tomorrow. That's a nice sound bite. I love it. Thank you, Mark. And thank you, everyone, for joining us. And we will all see you soon on the next OpenShift Commons briefing. Thank you. Thank you. And this is when Chris is...


At the thought of family gatherings during the holidays, who can blame us when we prefer a negative peace, which is the absence of tension, to a positive peace, which is the presence of justice? Because the kind of peace which gets announced by choirs of angels is a messy, conflicted peace. It means wading into division and demanding a reckoning for every way in which we have diverged from the requirements of love. The peace which passes our understanding doesn't lead to many silent nights, but it's the only way to that new and glorious morn.


It's quite remarkable, isn't it, that in this complicated machine called the body, the core of it, that which helps propel the rest is still fraught with mystery. The center is dedicated to the task of removing as much of that mystery as is possible. And it will succeed. And there will be breakthroughs. The new Javad Mofagi and Center for Brain Health is really going to exemplify this partnership between research and patient care. It is inspiring to see that in a world with increasing uncertainty, there are still places where concrete projects are being built that will contribute to the advancement of knowledge and our improvement as a society.  When they have the opportunity through their formal medical education to actually be part of the research process, to actually interact with patients and their families, that we will end up with an even better medical graduate.


Welcome back for part two of my lecture on integrative modeling of biomolecular complexes in the context of the Bar-Excel online summer school. So now that you know everything about docking in general and docking with HADOC, we're going to move into application example illustrating how you can use diverse set of data and diverse information to guide the modeling process. And we'll start with the modeling of antibody antigen complexes. As you are probably all aware, the antibody structure consists of two chains and heavy and a light chain. And the action is happening in this head where you have the, you see here the light chain in magenta and in more grayish color, the heavy chain. And the loop in these regions are the loops that specifically recognize their antigen. They are consisting of six loops. So we are, they are called H1 to H3 for the heavy chain loops and L1 to L3 for the light chain loops. H3 is typically the longest and the most difficult to model because of conformational viability. So this is where the binding usually takes place. And this is of course information, this is knowledge that we can use to guide the modeling process. So to benchmark actually how well this kind of information can guide the modeling process, we use the small data set of 16 complexes that are extracted from the docking benchmark 5. So this is a reference in the field when it comes to docking. This is what you are using to basically measure how well your method is doing in terms of predicting complexes. We decided to test different docking approach here. This is the work of Francesco, former PhD student in my group. And we use four software which allow first that are simple to use, that some of these are available as web portals, but they also have specific options to deal with antibodies. So CLOSPRO, ZDock, these are two rigid body grid based fast forward transformation docking software. And they have options to either define these hyperviable loops or mask regions for the search. And LightDock is one software which uses artificial intelligence, swarm based optimization, and also has options to define residues that are important for the interaction and our own Haddock software. So those allow you to define restraints either for the scoring, which will be done more in CLOSPRO and ZDock or to drive the modeling process which is done in Haddock and LightDock actually. So we tested different scenarios which represent different types of different levels of information on the system. For the antibody it's clear we have the hyperviable loops where we know that the binding is taking place. We might not know exactly which amino acid within these loops is actually contacting the antigen, but in general we have this information. On the antigen side things are more complicated. It's not always easy to predict where things are binding. So we have actually three different scenarios. In one scenario we assume that we have no knowledge of the binding site on the antigen so we are targeting the entire surface. For Haddock this will mean that we define all solvent accessible surface residues as passive. And then we have a scenario where we say we have some knowledge of the binding site on the epitope and this knowledge is based on the knowledge of the crystal structure. So it's an artificial set. But we define all residues that are within 9 angstroms of the antibody as being part of the binding site. So that's a rather well defined epitope. And then we have the ground truth which is all the residues that are making contact at 4.5 angstroms so this is the perfect scenario. So you have a perfect definition of your binding on the antigen side. So we tested all three scenarios with all four software. How do we measure the quality of docking models in general in the fields? So these are criteria that are used in CAPRI, this critical assessment of predicted information, well accepted in the field. So we have three measures. The first one is what is called a fraction of native contact. So if you have a reference structure, a crystal structure, often you have a list of contact that are made across the interface. If you have your docking model, you can calculate how many of the contact that we find in the reference are present in the model. And this defines this fraction of native contact. So the number of common contact between the two divided by the number of contact in the reference. So if you have a fraction of native contact of one, you have predicted all contact correctly. So the other measures require superimposing the model to the reference and then we have two measures. In one case, we superimpose on the receptor, the largest of the two molecules. And then we calculate the RMSD difference on the ligand, on the smallest of the two. So that's called the ligand RMSD, error for ligand. The other one, we do the fit of the molecule of the model onto the reference on the interface. And we use for that the interface of both the receptor and the ligand. So this is more focused on the interface. Because it's more focused on the interface, the RMSD values that you are calculating are typically smaller. Here the RMSD values are larger because changes which might be removed from the interface might lead to higher RMSD value even if the interface is well defined. So to define a high accuracy prediction, we want to have at least 50% of the contact correct. The ligand RMSD or the interface RMSD should be below one extra. We have a medium quality solution which is attributed to two stars in capri terminology. If you have 30% of the contact correctly predicted, ligand RMSD below five or interface RMSD below two. And you have an acceptable solution if 10% of the contact at least are correctly predicted. And the interface RMSD is below four or ligand below 10. So this is the matrix that we're going to see in more cases in my examples. And here are already the result of all the docking that we've been doing. So a free scenario times four different software. Each row in this plot corresponds to a scenario. So here no information on the antigen. So we have the entire surface of the antigen. Here we have a loose definition of the epitope on the antigen. And here we have the true interface. So this is the perfect case. Each column corresponds to a different software. What do you see on the x-axis of those plots? This is the number of models that have been ranked by the software which you are considering. So what's your success rate? Which is the y-axis. If you only take the top one model predicted by the software. And we depict the quality of the models by colors. So dark green is high quality within one X-Trum. Light green is medium quality within two X-Trum interface RMSD. And blue is an acceptable model. So if you look at all the plots together, you directly see that actually the header column has the most dark green. And also has in general the highest value of all. So that's the first observation. So using the information directly in Haddock to drive the modeling process does benefit clearly the generation of models. What you also see is that the more information you input into the system or the better the information is that you have the better all the software are performing. So here you have the true interface. And you see that Cluspros, ZDoc, they are all doing very well. Actually all software are doing quite well if you select the top 100 models. But not all have the same scoring capabilities. For example, a LightDoc is doing very well even without information on the antigens. So you see a success rate here that goes to above 60%, 65% success rate in the top 100. But it's not able to identify those models because they are not scoring in the top say 10, which is only 6%. Haddock will reach about 31%. If you have the entire surface, top 10, which is actually slightly higher than Cluspros and ZDoc is reaching about the same level as well. As soon as we start adding information, then you see that things change. So we see Cluspros is also all software benefit from the information. But we see that in general actually Haddock is one of the best performing and we see that we are generating now high quality model because we use the data directly to drive the modeling which is not the case in Cluspros and ZDoc. And if you have the perfect interface then you have 100% success rate in case of Haddock, even top 1 model while the other software reached out after top 10 or top 15 in the case of ZDoc. So there's a clear benefit in using the information for the modeling process. So these were single structure based statistics. So now we can look at cluster based. So if we do clustering and we assess the clusters and you see here this is only Haddock in this case, the three scenarios, entire surface, loose definition of the epitope, perfect definition and now the X axis goes only to top 5 because the clustering limits very much the number of models that you have to loop and you see that after clustering if you have good data you reach 100% success rate. This was already the case in single model. With a loose definition of the interface we are above 50% with clustering if you look at top 1 and if you look at single structure clustering we are below 50%. So clustering does help also in the scoring but if you look at here, single model, no information on the epitope we have 25% if we look at single structure and we are below that if you look at cluster based because there might be more difficulty to cluster structure where you have very little information. The message here is if you have no information at all it's better probably to do a scoring on a single structure based for Haddock if you have information then clustering really is the way to go. So if you want to read all the details about the story and many more things you shall refer you to this structure article that was published last year. So using information and in this case for the antibody it's the knowledge of the hyperviable loop with some knowledge of the epitope if you have it clearly improved the modeling of antibody antigen complexes. So let's move now to another application example which in this case is modeling from mass spectrometry data. And this is the work of Adria Melchion from a post-document group and we are going to look at an assembly, a complex which has to do with the bacterial circadian clock. A circadian clock is the reason why we have jet lag if you move say from Europe to the US. So we have an internal clock that kind of measures day and night. And bacteria also have some bacteria also have such a system. And we are here looking at a cyanobacteria which actually uses light as an energy source. And it has a very fascinating system. So you only need free protein in this bacteria to generate this molecular clock. So you can overexpress those free proteins called K, A, B and C. You add phosphate and ATP and the clock starts ticking. That's all you need. How do you know the clock is ticking? You can monitor, there is a phosphorylation, dephosphorylation process taking place between these proteins. And you can monitor the phosphorylation state of the system by mass spectrometry. And then you can really follow the frequency of your clock. So MS in this case gave us some information. For now we are only going to focus on the complex form by K, B and K, C. So from native mass spectrometry where we are looking at a full complex we know that the stoichiometry of the complex should be 6 to 1. So 6 K, B molecules binding to 1 K, C. And doing hydrogen deuterium exchange experiments with MS detected by MS the binding interfaces of those proteins have been identified. So you see here K, B which is the smaller of the two proteins and the blue regions are protected from HD exchange when the complex is formed. And this defines one surface of your protein. We also have some mutagenesis data, those free amino acid here, arginine and lysine, so positively charged amino acid if mutated, abolish or alter the binding properties. Here you see K, C which is the larger of the two components. So you see it's a much larger system. It's kind of a double donut. And what you see in blue are again here the HD exchange data identified by MS. So those regions are protected from exchange when the complex is formed. And you see that there is a 6 volt symmetry. So on the top you identify 6 binding site and at the bottom we also identify 6 binding site. So this will be 12 in total but we know that the binding is 6 to 1. And we also know that the binding should happen either on top or the bottom but they should not be a mixture. What is also interesting is that if you open the donuts you see that there are also protection differences at the interface between those two. So there seems to be a communication between the top and bottom region of the system. So what did we do? Since we have two binding sites, we did two docking experiments. One is targeting this one. So we didn't dock 6 to 1 but we docked 1 kb on 2 kc and we only docked on 2 this region. And we did the same for the bottom. So the top region is called the C2 region and the bottom region is called the C1 region. And this is the outcome of the docking basically. So we get two sets of solutions. Based on the Hadox score we could not say if the top solution C2 were better than the bottom solution C1. Forget about this for the time being. And these are the different clusters that we obtained. So we have two sets of solutions and based on our scoring function we are not really able to distinguish those two. But MS comes to the rescue and in this case it's ion mobility mass spectrometry. So in mass spectrometry you can have a molecule moving in a spectrometer so the native complex moving in a spectrometer and you're measuring the time it takes to this molecule to bridge a fixed distance in a spectrometer. It's flying against a gas flow of ions. And the hydrodynamic properties of the molecule will define the times intake. As if you are swimming in a swimming pool, if you put a sombrero hat and go swimming you're going to swim much smaller, much slower than if you have no hat. So this has to do with your hydrodynamic properties. So by measuring this time of flight basically you can get information about the 3D shape of the protein. So that's a long extrapolation but it's an experiment which is quite easy to do. This has been done to study the maturation of viruses for example but it's also telling you about the arrangement of a protein. If you want to read everything about it I refer you to this Nature Protocol paper 2008. So those data were measured for this KBKC complex and this brings us now to this value that you see here below those complexes. So those values are the value that we predict based on the model so we can back calculate what this collision cross-section is and it's a surface area, you see the units are nanometers to the square and we can compare those predictions to the experimental values and the experimental values are indicated by the dotted line here. So the experiment tells us that those values should be between 133 and 140 square nanometers and if you see the different solution you directly see actually that this indicates that the C2 solution should be fitting the data better and the C1 solution is actually has two large surface areas for the experimental data. So based on this we predicted our best scoring model of the C2 interface as being representative of this complex and this was published in 2014 in PNAS. Now in 2017 our collaborators managed to actually get a cryoM structure of this complex and what the cryoM model reveals is that C1 is the right structure. So we screwed up and this happens, you know, in research it's going to happen to you for sure, at least once, it's part of research. So we should not hide this kind of failures but we should try to learn for those failures and there are different reasons here why things went wrong. One possible reason is that we are dealing with a non-globular system. If you look at KC, you see this protein as a hole in the center. So it could well be in the spectrometer when you do MS those proteins are flying in vacuum. So if you have something which is non-globular, it could well be that the system compacts in the vacuum in the spectrometer. And if this happens, what you're going to measure is an underestimate of what you have in solution. And this will basically, so if this is an underestimate of what you have in solution you have to move up those two dotted lines and then the green one becomes the more, the better fitting one. So that could be one explanation. But there is another explanation here. And this explanation is that nature is also fooling us. When we did the modeling in 2014, we used a crystal structure which was a perfectly fine crystal structure of KB. When the cryo-EM complex was published, there were actually two articles back to back published in Science and one of those was describing a crystal structure of KB but that structure was different. The fold was different and this different fold is also what is found in a cryo-EM structure. So it's exactly the same sequence, the same construct but if you compare those two crystals structure so the one that we used when we did our initial modeling was this ground state fold this GS fold and you see it's beta-alpha-beta, beta-alpha-alpha-beta The same sequence later on was crystallized in a different fold the first part is the same but the second part is completely switched so you go from alpha-beta-beta-alpha while it was beta-alpha-alpha-beta before Same sequence to different folds so this is not something that you can really predict we had a crystal structure, we trust that crystallography is generating good models and there was nothing wrong with that structure but this is one of those rare examples where a sequence can exist in different folds so now if we take the correct fold of that protein and we repeat the docking and in between we also had improved our modeling capabilities in Haddock by introducing coarse-graining meaning that we are going to simplify the representation of the system so we group four atom, four heavy atom into one particle so we have less particles to deal with so it's a smoothening also of the energy landscape of the surface and for this we use the martini force field from the Silver Tians marine group we do the docking at this coarse-grained level and at the end we transform back the system into a fully atomistic description and we have support for both protein and nucleic acids and this was described in those two publications so we use this approach and now we want to model the full complex we have six kb binding to kc so it's a seven molecule docking experiment that we are doing we use the same data that we used in 2014 we apply c6 symmetry because we have this symmetry in a complex and seven body docking using the coarse-grained implementation in Haddock and what you see here are the docking models superimposed onto the crystal structure so it's not perfect but you see that there is actually quite a nice similarity not the crystal of the cryo-em structure which was about 4.5 angstrom if you look at the center of mass of the crystal of the kb molecule in a model it's very close to the center of mass in a crystal what is even more interesting is that now the correct solution which is the bottom solution scores much better than previously so previously we could not distinguish between those two in addition by doing coarse-graining we have a 7-fold speedup in the docking process we did not use the cryo-em density for guiding the modeling process but we did use it to validate our model now if we fit our model to the cryo-em density and we use camera to do that the correlation score of the model was 0.82 the correlation score of the structure deposited in the PDB is 0.84 so the model is slightly lower than what has been experimentally determined but we did not use the data in the modeling process so that's a nice example of using first the right structure, the right fold but this is not something that you can really foresee and using an improved methodology, coarse-graining that allows us to model the full assembly leads to the correct solution so that was for us a lesson since I was mentioning cryo-em now it's a good way to move into the cryo-em topic so cryo-em and cryo-electron microscopy and cryo-electron tomography now are really the new stars in structural biology so you're probably all aware of what's happening there so you are basically vitrifying a sample and then you're making an image with electron of this sample and in the recent, in the last years there has been a huge development in cryo-em because the detectors in particular have been becoming way better and because of that cryo-em has moved into higher resolution structural biology so I think the record now is probably 1.6 extra resolution so what you're getting are 3D reconstruction basically so you have 2D images that you have to interpret find what the orientation is that you are looking at and then you are reconstructing a 3D object if the resolution is high enough then you can directly fit your proteins and you build your amino acids even in density as you were to do for crystallography but there are still cases where the resolution might not always be sufficient and in those cases you would have to rely on some over-modeling tricks so this was the resolution typically before 2013 we were speaking of blobology where you will have maybe 10 extra, 8 extra resolutions or even 30 extra where you can see the shape of your molecules shape of your complexes but you cannot really distinguish which amino acids go where and this day we have full atomistic picture so that's really an impressive improvement and for that they also receive, the technique, receive the Nobel Prize now if you don't reach high resolution enough to build the model and this is what has been happening for many years you need to, so what has been done is to use existing structural information so crystal structure of component of a complex and then try to fit those into the density to generate a 3D model of your complexes and this is still happening because not all cryoEM maps reach the resolution which is sufficient to build from scratch also you should know that resolution in cryoEM is not a constant resolution depends on where you are in the map so some region of the map may be very high resolution and some of the regions might be low resolution so you might be able to build from scratch structure in part of your maps and you will have to rely more on modeling for other parts of the map now how this fitting was done or is typically done is to do that one molecule at a time so you take one molecule of your complex you search a bit like we do in grid talking you search all translation and rotation and you measure the correlation to the electron density in that case and you find the locations where those molecules fit but you do that one component at a time meaning that you never take into account the interactions between the component typically during the fitting process so you don't take energetics into account and flexibility is usually added afterwards so we want to see if we could actually use Haddock which used the energetics during the docking process to improve those cryoEM models to generate better more reasonable views of those interfaces and this is the work of Hido von Zundert who implemented basically EM restraints into Haddock so the way we do that so Haddock uses CNS CNS is a crystallography NMR system it's a software used in NMR and in crystallography to calculate structure of molecules it has a lot of restraining functions so the distance, this ambiguous restraints that we are using in Haddock are actually applied in CNS CNS is also a way of describing density because it's used for crystallography so you can transform an electron density into a crystallographic representation and then we could use the energy functions that are in CNS to optimize against this density now if you try to dock directly against the density the system is not converging so there's a bad convergence and it's not working so our way of solving this issue is to first identify in the density the most likely location of the molecule and replace centroids at the center of masses of those and there are ways of doing that to come back so once we know where the molecule should end up you might not know which molecule end up where we define a distance restraints from the center of mass to each molecule to those centroids and we used distance restraints again into Haddock to bring the molecule together together with electrostatic van der Waals interactions so this is basically generating an initial model of the complex once this complex has been generated we turn on the energy term that represents the density and we're going to optimize against this density and these optimized solutions are going into the flexible stages of Haddock where we also have these energy functions active so we can now incorporate EM data into the modeling process into Haddock and combine this information with any other type of data that we might have to drive the modeling process now how do we get those centroids since there were ways of doing that but we wrote our own fitting software which is called PowerFit and PowerFit gives you next to the location of the molecule and really it outputs also the coordinate of the centroids the most likely location of the center of mass of the molecule this is the input that you can then fit into Haddock so here is one application example so we're looking now at a 16S ribosome and we're looking at a protein called KSGA binding to that ribosome there is an EM map available this is the resolution 13.5 angstrom we have the crystal structure of the ribosome we have the crystal structure of KSGA on the RNA side there are data from Hydroxy radical footprinting that are telling you where the protein is binding so basically the RNA is protected from these Hydroxy radicals because of the presence of the protein so this is pointing to the binding site on the ribosome site and on the protein side we have also mutagenesis data so there are three residues that have been mutated and shown to prevent the interaction, prevent the binding basically so this is all the information that we have to do this modelling process of course there is a structure, a model of this structure of this complex that has been deposited this is for ADV, it looks beautiful as long as you only look at the backbone representation but when you turn on the side chains and you look at the interfaces you see a lot of clashes because the fitting was done in a rigid body fashion so everything yellow here are clashes if you look at the three mutations that are important for the interactions one of them connects to the RNA but it clashes actually and the other two are not even contacting actually only this one makes a reasonable contact this one clashes and this one is not even contacting so the model that was built based on EM data does not really explain the data that we have on this system so can we do better? that's the question so can we use Haddock to generate a model which will not have all these clashes, all these bumps and here is the result of this modelling so you see here now the RMSD with respect to the deposited structure in a PDB versus Haddock score this score now contains the correlation between the model and the electron density you see that we are getting basically one set, a unique set of solutions and this is the solution that we are getting and now you can zoom in in this interface and look at for example at those three amino acids that were mutated and they are all making nice interactions it doesn't per se mean that they are correct but at least they explain the reason why if mutated those residues are preventing the binding and what is also interesting is that the model now reveals additional key residues there are two arginines that we identified from the model that seems to play an important role in the binding as well those two, these were the other one so this gives you additional handle to test the model so we could do mutagenesis on this and test if indeed this is changing the binding and all those amino acids that were previously mutated but also the one that we identified also actually quite highly conserved so you see a map of conservation on the surface on the backbone of the protein so this is all consistent so we have now a way of supporting cryo-electron microscopy data in Haddock it's available also from the web portal and it can be combined with any other type of data that we have with Haddock so now last part of my story we are going to move into membranes and membrane complexes in particular so about 20% of the protein is consist of membrane proteins and if you look at the statistics so you see that the number of say the number of proteins, unique proteins and the number of membrane proteins is very different so this is statistic from the PDB so we have a lot of structure in the PDB that are say soluble proteins and this is a small amount of membrane protein this has been increasing in recent year because this is not going on the way to 220 so the situation is becoming better but a lot of the new structures are all of the same type these are GPCRs a lot but those membrane proteins are very important because about 50% of the drug targets are against membrane protein and they have their challenges to study experimentally difficult to study by NMR, by crystallography and by cryoEM you start seeing models of the membrane proteins but the resolution in the membrane might be sometimes problematic so you also need methods to be able to model the complexes that are formed in the membrane but if you think of what's happening in the membrane the environment is different so inside the membrane we have a very hydrophobic environment in water, for the soluble protein we have of course a water environment and when a complex forms in water typically what you observe is that the regions or the interface regions are quite hydrophobic so you want to bury hydrophobic regions when you are in solution in the membrane this is not going to give you much discrimination so in the membrane the membrane will have will impose some kind of different energetics to the recombination process and further the membrane also impose restrictions because you know that the protein cannot rotate 90% 90 degrees to be inside the membrane there are regions of the protein that want to be in the membrane and there are regions that want to be outside the membrane so this is topological information which in principle we could use and pretty much most docking approaches that have been developed until recently were only targeted and optimized for soluble proteins there are some examples of there is some work from different groups on using membrane potentials so first thing we did was to define a benchmark, identify complexes it's a small benchmark about 35 entries but unique proteins so no overlap between those entries and we have tested HADOC we bought any just in a standard protocol basically we bought any membrane specific tricks and the data set is available from SPGREED the data repository of SPGREED so if you are into scoring function optimization you could download those and try to optimize things so if you have data to guide the modeling HADOC is doing already quite well and in the iron parts example I showed you at the end of my first part you see that you can do very reasonable docking using the knowledge of which loops are in the solution still we wanted to explore more this field and one question that we wanted is can we actually explicitly account for the presence of the membrane in the docking process and for that we combined two docking approach I only mentioned LightDock which is a swarm intelligence based docking approach with HADOC and this is the work of Jorge and Brian and Brian and Jorge developed previously LightDock before joining my group and now we have been integrating the two so LightDock is based on a glow worm swarm optimization algorithm and glow worms basically you have this energy landscape here and glow worms attract each other depending on the amount of light they emit for docking purposes light is transformed into some kind of scoring or energy function and glow worms they are usually come in pack in a swarm so what you have is that you have a lot of different glow worms you see here two swarms so this is one swarm this is the other swarm and the glow worms will tend to move in a direction attracted by the over glow worms that have lower energy so this guy here is going to move in the direction of this guy because this one has a lower energy than this guy it's also similar to what you see you know flocks of birds flying they follow very closely each other the same is going to happen here but in a complex energy landscape we use that to sample this energy landscape now LightDock also has the ability of defining specific residues important for the interaction so LightDock was used in this work so how this does protocol for membrane docking works so we take so since we are using crystal structure from a PDB there is a database called Mempro MD where they have been doing embedding those membrane proteins into a lipid bilayer using a coarse grain opposition in the martini force field so you can find the website of this database here so we extract the receptors from this database together with their membrane and what we keep are only the phosphate groups of the membrane and these phosphate groups are going to define a boundary for the docking process now if we know that we are targeting the extracellular loops for example we are going to position the initial swarms so this is one swarm around those loops so each blue point here is a swarm of glowworm basically and these glowworms are basically different starting position of the ligand conformation that we are going to dock and then we run the LightDock optimization and the models that come out of this optimization are clustered and then because LightDock is more of a rigid body type docking approach it has a flexibility option as well we need to refine those models to remove clashes and for that we are going to use Haddock actually so that is the pipeline these are all the complex that we tested so out of the benchmark these are the complexes that are where a soluble protein is binding to a membrane receptor and you see here the results of the LightDock docking in gray will be the docking results without accounting explicitly for the presence of the membrane and explicitly means using those phosphate groups and you penalize solutions that penetrate this phosphate layer basically once we use this membrane topological information into LightDock you see that the results improve dramatically in all cases so these are the alpha helical complexes, beta-barrel, antibodies and if you took that this is the all complexes together and again you have the number of models that you consider so great improvement by using the topological information of the membrane now those models typically have clashes at the interface so what we're going to do is to use Haddock and we use the coarse-drain to atomic transformation of ability of Haddock to basically refine the model so we don't do docking in Haddock, we just use Haddock as a refinement tool and what you see here is again all the complexes with the left column being the models ranked based on the score and the color coding indicates the number of clashes so you see the red one has more than 100 clashes at the interface so all the rigid body, all the LightDock models have quite a large number of clashes after Haddock refines you see that they're all transforming into majorly green colors meaning that we have removed the clashes without modifying much the structure of the complex itself so we have this new protocol that combines LightDock and Haddock and accounts explicitly for the presence of the membrane and we declash them all at the end with Haddock so this was actually published in Nature Communication earlier at the beginning of this year actually you have the bio-archive reference but it's out in Nature Communication now in terms of perspective, Haddock the machinery itself can already handle explicit membrane so we can do docking with explicit membrane and this was demonstrated in different papers so these are users that have been using the server we needed to change some parameters to allow for that but once we had done that they could do docking in this work they are doing a study using NMR data to look at the binding of a protein on the nanodisc and this is a simple test case that we did of a small molecule docking into an ion channel with explicit presence of the membrane so we might be moving toward explicit membrane docking in the future this is something that we are actually benchmarking and developing now so that's work in progress so now to conclude and give you a little bit of perspective so I hope to have given you an overview first of the docking methodology in general and convince you that using data when you have them using information to guide the modeling process is really a useful thing to do you have to realize that when you do this kind of modeling what you're getting out are models and these are not experimental structure but the models are very valuable to generate new hypothesis and drive the experimental work so you can build on the model to validate those as such information-driven docking integrative modeling is very complementary to classical structural methods so where is the field going? so what you see here is a picture that we created actually to celebrate the 50th anniversary of the protein data bank and this is an illustration of what I would like to call integrative structural biology of dynamical landscape so you see this funnel where you have a cryo-M machine, mass spectrometer, NMR machines you might be doing large-scale experimental assays you combine that with software and together you get a representation of macromolecular assemblies but we'll have to capture not only one model but many of those assemblies are dynamical so I think the trick will be in the future to also describe the full landscape of those assemblies you see here the 50 has different conformation it's more open here, it's more closed here so we will not have only a single representation of those structures but we might have a movie of those and that's I think the challenge that the field is moving toward now these kind of models are not usually accepted in the protein database but the protein is working on ways of representing those models and what you see here is the PDB dev which is a prototype archiving model for integrative models so if you are working in a field and you're generating some model you can't deposit those models now and in the future this will become integrated into the PDB with that I want to close I want to thank the people in group who have been contributing to many things that I have been telling you today so this is our usual group picture since COVID is here we have support from different national and European projects and of course Bar Excel is very key to all the software development happening around Hadock and other pictures of former group members and software developers that have been contributing some of those you have seen already in my slide and thank you very much for your attention