A very good afternoon to all of you. My name is Sohail and I work for Airtight Networks. I'm a WIFI security researcher. And today, I'm going to talk about a security problem in WPA2 protocol. You may have heard about whole 196. In fact, in the last one and a half week, a lot has been said and discussed about whole 196 or GTK vulnerability. And in fact, a lot of speculations have been seen around this problem. So today, we are going to present the real picture and we'll help you understand or see the implication of the problem. So let me answer the most frequently asked questions that people have asked to me. Where did we find this whole 196? And for your information, it's already there in this standard. For the last six years, it was buried in the standard on page number 196. And that's the reason why we are calling it a whole 196. If you see the last line of the page 196, the designers of dot 11 have warned about the weakness of WPA2, one key that is used in WPA2. And on the last line, there is something mentioned about that key. GTKs do not have this property. So designers are talking about some property of encryption key, which is not there, which is lacking. But this problem was never discussed. And in fact, there is a problem, but it was never brought to the attention of WPA2 users. So in this presentation, we'll talk about that. So this talk is not about unauthorized user taking or gaining access to your corporate or your trusted network. No, it's not about that. It's not about key cracking, OK? It's not about cracking private key of a user. This talk is about an insider attack that can be carried by a malicious user who is already part of your network, who is already inside the network. But I have seen people saying that, OK, when a malicious user is already inside the network, then he can do pretty much anything or everything in that network. OK, so what's a big deal? Fine. So let me ask this question to them. If, let's assume that if you are an analysis insider, can you really sniff private data of other Wi-Fi users? If you try to capture packet in the air, you will see encrypted packet. And since you don't have key, you will not be able to decrypt it. So can you really sniff or see private traffic of other Wi-Fi users, even if you are inside the network? And the trick to do that is basically one can try launching man-in-the-middle attack. And the most popular trick is basically in Wi-Fi network is setting up a honeypot, OK? Using this, one can actually do man-in-the-middle and try to capture all the traffic of a user. But in a WPA2 networks, client devices are actually configured to connect to WPA2, OK? So those clients are not going to authenticate honeypot access point or they will not connect. So this trick is also not going to work. So what are the other options available? So probably we can think of launching man-in-the-middle attack in a very conventional style, old style, with the help of ARP spoofing attack. But today's network are basically much more secure than what it used to be. There are security systems which can actually detect ARP spoofing attack, wired IDS, IPS, or even switches have capability to detect it, OK? So if you are going to launch ARP spoofing attack, the chances of getting detected is very, very high, OK? So now let me ask again this question that without getting detected, can you really sniff private traffic or data of other Wi-Fi users? Even if you are inside that network, is it really possible? The answer is yes. In this presentation, I'm going to show you that it is possible. And this will not involve any key cracking or brute force. How? Let's walk through the presentation and we'll see. So I'd like to briefly talk about WPA2 protocol. I'm sure many of you must have a good technical understanding of that. But just to recall, in WPA2 protocol, there are two types of keys which are used for traffic encryption, packet encryption. One is called PTK, pairwise transient key. And the other one is GTK group temporal key. And the purpose of PTK is to protect unicast traffic, OK? While the purpose of GTK is to protect broadcast or multicast traffic. So there are two types of keys. If you see PTK is a unique key. It is unique for each user. And it is derived per session. So for example, if two clients are connected to a WPA2 enabled access point, both of these clients will be using two different keys. So client one will not know the private key of client two and client two will not know the private key of client one. So their traffic is basically secure. But GTK is a shared key. It is shared among all associated client devices. And it is actually managed by access point. So whenever a client connects to access point, it receives a copy of GTK from access point. So at any point in time, all associated clients are going to have same copy of GTK. Now I said GTK is used to protect broadcast traffic or multicast traffic. So in a Wi-Fi network, if you have noticed, all broadcast traffic is basically sent by access point. Because access point can talk to each and every client. And this traffic is never sent by a client device. Because client can, whenever a client has to talk to other Wi-Fi client device, it first send packets to access point. So for a client, destination is always fixed. Which means client always send unicast packet. And that gets, so does that mean client never generate broadcast packet? It generates. But in dot 11, that broadcast frame gets mapped to unicast frame. We'll not talk about that. But just to, so which means by design, GTK is going to be used as a one way key. Access point is going to use it for traffic encryption, broadcast traffic encryption. And all client devices are going to use it as a decryption key. And that's how the software in access point and clients are programmed. And here I'm showing you a log of Wi-Fi client device. There is a software called WPS supplicant, which runs on client devices. And it shows that client has a copy of GTK available, right? So the group key is always available to client device. It is always known to client device. Why it is available? Why client keeps a copy of GTK? Because it has to decrypt packet coming from access point. Now let me ask, what is stopping that client not using that GTK for its own packet encryption? So protocol says something which is a normal behavior. Client should not use this key for packet encryption. It should always use this key for packet decryption. But analysis user can always exploit this key. Analysis user present in that network can always use this key to create its own packet, generate its own packet, which means for analysis user, one broadcast frame is available, which that analysis insider can generate, is created, and send that packet to all other Wi-Fi client devices. But Wi-Fi client devices are programmed to receive packets from access point, not from other Wi-Fi client device, right? So in order to successfully deliver that packet, this insider is going to spoof the MAC address of access point. So once the attacker keeps the MAC address of access point as a sender of that packet, all clients present in that network and associated with the same access point is going to receive that packet and going to decrypt it successfully, right? Which means analysis insider has access to a broadcast frame, which it can create and generate and send. And there is no way for other Wi-Fi client devices to differentiate whether this packet is coming from other analysis user or it is coming from access point. Absolutely there is nothing present in that packet which can help a client to, you know. So now it's clear that attacker can send broadcast packet. So he can do a lot of attack. It depends what kind of attack payload he's going to encapsulate inside that broadcast frame, right? So in this presentation, I'm going to show you three exploits of GTK and the first exploit is basically using the ARP payload to encapsulate and create broadcast frame. So in this example, attacker is generating its own broadcast frame and sending that frame to other Wi-Fi users present in that network. And basically the attacker is advertising that he's a gateway of that network. All other clients are going to receive that packet and update its ARP cache thinking that the gateway, this new machine has become the gateway of that. And that's how our protocol works.  So attacker becomes the gateway of the network for that client who receives that packet. And after that, all traffic generated from that victim is going to be destined to this new gateway who is the attacker. So victim sends all its packet to access point, encrypting its packet with its own private key which is known to this user or access point. So access point will be able to decrypt those packets. And at this point, access point finds that this packet is actually destined for other Wi-Fi user who is also part of the same network. So access point forwards that packet to other Wi-Fi user. And that other Wi-Fi user happened to be a malicious attacker, right? So finally, the packet reaches to that insider. So just by injecting one spoofed or gratitude USR or ARP request frame, the attacker is able to redirect all the private traffic of a legitimate user in that network to its own machine. At this point, attacker can actually drop all the packets and do a denial of service attack in the network. But he wants to steal information, right? So for that, he has to provide a seamless service to the user, a transparent service to the user. User should not feel that there is something wrong going on in that network. For example, if I'm accessing Yahoo, and I'm not able to access that, it means there is something wrong in that network. So what happens at that? Attacker is going to forward all that packet to actual server or gateway, and he becomes man in the middle, right? So just by exploiting the GTK shared key and forging its own broadcast packet, attacker is able to redirect private traffic of other victim to its machine and able to provide services. Since attacker is sending a broadcast frames, it means all other Wi-Fi client devices which are also present in that network are going to receive that packet and update its ARC. So attacker is not just victimizing one Wi-Fi user, he's victimizing almost all Wi-Fi user who are in the range of that attacker. Now one can ask what is the difference between this and the old style, conventional style of ARP spoofing attack? Okay, one can, I mean no matter what, I mean no matter if you are using a WPA2 network, I can always use ARP spoofing attack. I can always launch ARP attack to do man in the middle. So in this old style ARP spoofing attack, your packet first goes to access point and after that access point decides and forwards those packets to wired segment of the LAN or wireless segment of the LAN, which means the security systems present on that network wired IDS or IPS or switches are going to see that packet, which means the footprint of the attack is available on the wire and in any wired security systems which are capable of detecting ARP spoofing attack are going to capture that packet and detect this attack. But then when attacker is using this Stealth mode attack, packets are directly sent by one malicious user to other Wi-Fi users in that network. Packets are never going to access point and it is never reaching to wired segment of the LAN. Which means the footprint of attack is limited to the air, which means most of the wired IDS, IPS systems are going will not detect this because packet is not hitting the wired segment of the LAN. So without getting detected, attacker can do man in the middle attack and steal private information of other Wi-Fi users in a WPA2 network. Okay? So this is one example of GTK exploit or whole 196 exploit. I'm going to present here the second exploit and it's all about changing the payload, what kind of payload you are encapsulating in that broadcast frame. So this time attacker is going to use unicast IP payload inside that packet to basically launch IP level targeted attack. So for example, if frame is, at layer to frame is broadcast, but I can always use IP layer, layer three unicast frame inside that. And one example could be port scanning. If I'm going to use a payload, which is used for port scanning, then my packets will actually reach to the victim and victim will respond to that packet. Which means I can do port scanning and I can find out whether that Wi-Fi user is vulnerable or not. Which can further be exploited. So just think what else is possible. I mean if I know that this payload is going to do cause some buffer overflow, then I can inject that packet, right? I can probably, I can also inject malware. So a frame is available in which attacker can inject, in which attacker can encapsulate payload of his choice and cause serious damage to other Wi-Fi users. This is second example of GTK exploit. The third example of GTK exploit is some, is related to the replay detection in WPA2. There is something called replay detection in WPA2. So all encrypted data frames have this, there is a field called packet number, which is a 48 byte number. And this number is going to be present in all encrypted data frames. And this number is used to detect whether a frame is replayed or not. And how that replay detection works. When client connects to access point, access point informs that client that this is the last consumed packet number. So you should use this number for replay detection. So for example last consumed packet number was 700. Client is going to cache that number, save that number. And after that client is expecting data frame having packet number greater than 700. So for example, if I'm replaying packet, which I have captured at the time when packet number was 600. And if I'm replaying that packet, that packet will not be accepted because client will find that the packet number is less than the last successfully received packet, which was 700. When access point sends packet, it will send with 701 because that is part of its traffic stream. It has already sent 700, so next packet will be 701, 702, and so on. So when access point sends packet with 701, client is going to accept that packet. And once the packet is successfully decrypted, client is going to update its locally cached packet number. So now the packet number becomes 701. And that 700 number, one will be used to detect replayed frames. So attacker is going to exploit this replay detection logic. Here what attacker is doing, he's injecting a very large packet number, arbitrarily generated number, because it is a 48 bit number. So let's take this example in which client, the last successfully received packet number was 780. At this point, attacker is injecting a packet because attacker knows the GTK, he can create its broadcast frame, he can put any number of his choice. So he's going to put a number, let's say 9,999, 949s, okay? Client is going to accept that packet because at the time of replay detection check, it finds that packet number is greater than the last successfully received packet number, which was 780. So the client is going to update its locally saved packet number. And the local packet number becomes 9,999. Access point has no idea about this, this change which has happened on the client device. So access point will send broadcast packet as per its own traffic stream. So the last packet which sent by access point was 780. So next time it will send 781, 782, 783 and so on. But let's see what is happening at the client device. Client device finds that the numbers are too less to the locally saved packet number. The last saved packet number was 9,000 and I'm getting 781, which means somebody's replaying packet. So client device is going to drop all the broadcast packet whose packet number is less than its locally cast packet number. Now imagine the implication of this. All Wi-Fi client devices who have successfully received attacker injected high packet number will start dropping all the broadcast packet coming from access point, which means the protocol which works on broadcast packet will not work, right? Two clients will not be able to communicate with each other because they will not be able to resolve ARP. So attacker by exploiting this replay detection logic, he's causing denial of service attack in downlink direction. And which can break few protocols. So this was the third example of exploit of whole 196 or GTK vulnerability. Now I will talk about the POC code that was written to exploit GTK. And I have used two open source software. One is WPA supplicant software and I've used it. There was no modification done, so it was used as it is. And then I have used open source driver, mad Wi-Fi. And the main modification was done in the driver part of the mad Wi-Fi. So before we actually see what kind of modification was done, let me quickly present the .11 frame here. When a client sends a packet to access point, it is called 2DS frame, okay? From client to access point, uplink packet. And when access point sends a packet to client, it is called fromDS packet or it is a kind of downlink packet and there is a field present in the frame which indicates about the direction of frame, whether it's going from client to access point or it is coming from access point to client, okay? But if you notice, there is not much difference between 2DS frame and fromDS frame. If you do a cyclic shift, 2DS frame will become fromDS frame. You move address three to address one, address one to address two and address two to address three it will become fromDS frame, right? And this requires three line of code to do that. And one line of code to change 2DS flag to fromDS flag. So in four line of code, you can actually convert 2DS packet into 4DS packet, fromDS packet. And then you need to select a right key to encrypt that packet because client always use unicast data traffic, so it always use unicast key. So six lines of code were added to choose right key. So it's just about selection in which data structure what kind of keys stored, right? So in just 10 lines of code, it was possible to exploit the GTK vulnerability. So now I will do a demo of man in the middle attack in which GTK vulnerability will be exploited. And after that I will talk about the counter measures. So in this example, the demo that I'm going to do now, I'll be using two Wi-Fi client devices. One client device is going to be the same laptop which I'm using for presentation. And then I have one more laptop here, which is running backtrack. And I'm going to use one of the tools available for launching man in the middle attack. There are a lot of tools available using which one can launch MITM. So for example, these are the tools, very popular tools which are used to launch man in the middle attack. So I'm going to use one of the tools. So the only modification, the only soft code changes that was done was in mad Wi-Fi driver. Everything is available open source. I have set up one access point here and you can see the SSI of that access point is corporate Wi-Fi. And if you go, you will see that this is a WPA2 configured, this is a WPA2 access point. So I'm going to access my attacker machine with the help of BNC software so that you can also see what's going on on the attacker machine. Okay, so my machine is accessible, so we can access this. So this is the backtrack running machine which I'm going to use to launch attack. Okay, perfect. So is this visible to people at the back? Or should I increase the font size? All my stuff is here. And I have two Wi-Fi interfaces running on this laptop. One is running on top of unmodified version of drivers. I will be using it to demonstrate something. And then I have at zero interface which is running on top of a modified version of driver. So let's, so I'm going to use SSL strip software to launch man in the middle attack. Okay, okay. So I'm able to successfully run SSL strip soft tool. Now, the next step is to basically poison the ARP cache of the client, right? And for that, I'll be using the modified version of driver. So, and let's quickly check the, what is the default gateway present on this subnet. So default gateway is running on <IP_ADDRESS>. And if you check the ARP cache entry, you can see the default gateway is running on 8.1 and the MAC address is basically the last two bytes is FE72, okay? And we are going to replace the MAC address, the IP and this MAC binding with the attackers MAC. So, and the Wi-Fi users, the IP address of users to which I'm going to victimize is using IP address <IP_ADDRESS>. So I have used his IP address and this is the MAC address of my machine and this is the IP address of gateway on behalf of which I'm sending this spoofed ARP request packet. Now let's go back and see what is the ARP entry. So if you remember the last ARP entry for gateway was, the MAC address was FE72. Now let's quickly check. And you can see the MAC address of the default gateway has changed. It has become 0AD3, okay? So far everything is working fine. Let's hope SSL history works as per the expectation. So I'm going to open some site. Let's open, let's say, Gmail, oops. Let's try running, sorry, okay. So it looks like I've not be able to successfully run this SSL strip, but I was able to poison the ARP cache entry of a victim. And just to demonstrate that my packets which I'm injecting is not actually flowing on the wire, I'm going to take a packet capture. Here you can see right now I'm sniffing on a wireless interface of my client. And you can see ARP request and responses, frames are being exchanged between my client and the attacker. The attacker was sending request frames and my machine is responding to those requests. Now let me quickly connect a wire. And we will sniff what kind of traffic is flowing on the wire. And here you can see there is no broadcast ARP request packet coming on the wire. The packet which I was injecting, those packets are directly going on the, these are the response which is basically sent by my client devices. So the footprint of attack is available only in the air, it's not available on the wire, okay. But if I, instead of using a modified interface which was running on top of a modified version of driver software, if I run unmodified version, what will happen? So let me quickly, so in fact I have demonstrated that my packets are not actually running, going on the wire. Let me quickly present all basically countermeasure techniques. The real fix is basically fixing this problem in the protocol and for that one can think of duplicating the use of GTK and group address data frame. The other solution is basically using the endpoint security software like SNOT or ARP watch which detects ARP cache, change in the ARP entry, but there is a limitation of this software. It is not available for all kind of Wi-Fi devices. Then one can actually think of using PSPF for client isolation feature. But then this can also break the legitimate communication between legitimate users. For example, if a user wants to share some pictures from laptop to smartphone, that kind of thing will not be possible. And we can probably not use this feature in a voice over Wi-Fi deployment scenario because then two Wi-Fi client devices will not be able to communicate which means this will create a problem. So can we really think of using wireless monitoring system? Since the footprint of attack is available in the air, it is possible to detect that in the air. So Vips can also serve as an additional layer of security to in detecting and protecting from such attacks. So the final conclusion is that all WPA2 networks are exposed to whole 196 and the inter-user privacy that was guaranteed is now broken in WPA2. The real fix is to fix this problem in the protocol. In the meantime, enterprises can think of using PSPF kind of feature, but for endpoint security they can use a SNOT or IPsec, but then you say it is not foolproof in some certain scenario it will not work. So multi-layer of security is required and probably over the air security can be used to detect attacks like whole 196 or any zero day vulnerability. So the final words, be aware, be secure. That's it.


All right, welcome, everyone. Thanks so much for attending. My name is Eric Maugham. I work at Pivotal on the core Diego team. And today, I'd like to give you an overview of what Diego is in the context of Cloud Foundry and an update on some of the things we've been working on over the past year or so. So first off, let me tell you a little bit about what Diego is. In essence, it's Cloud Foundry's native container runtime, the system that's responsible for coordinating containerized workload across to Cloud Foundry deployment. So just a brief history of Diego. We started development on Diego almost four years ago, January of 2014. And a year and a half later, we declared ourselves generally available, suitable for production use. We finally cut a 1.0 about a year ago. After we validated, we could run a quarter million containers in a realistic CF environment. And that started the clock on the deprecation schedule for the previous container runtime, the DEAs, which were retired this past May. So the ecosystem is now fully consolidating on Diego as the core container runtime for Cloud Foundry. So today, I'd like to give you an overview of how Diego and its components fit into Cloud Foundry and coordinate with the other subsystems to keep Cloud Foundry running, give you some perspective on how those core components are typically deployed and how they've been evolving over the past year or so. I'll then give a few updates on some of the things we've been working on in support of broader features across Cloud Foundry, including what Diego does in support of isolation segments, which you may have been hearing about, and some really interesting new security primitives that we call instance identity credentials that are enabling secure microservice communication patterns on the platform, as well as some other core features inside of Cloud Foundry that are improving stability. I'd also like to give you a brief introduction to some operator-focused tooling we've been building that we call CFDOT. And then lastly, I'll give a view into some of the next things we're considering working on for Diego in the next year or so. So to start, if you've been interacting with Cloud Foundry either as a developer or an operator, you're probably familiar with things like the Cloud Controller or the Go Router. And you've probably experienced the magic of running CF push and eventually seeing your Build Pack or your Docker-based or now your Windows application running on the cloud. So let me tell you a little bit, though, about what's going on under the hood here. All of these application instances are actually containerized, and they're running under the supervision of the component that we call garden inside of Cloud Foundry. This is effectively Cloud Foundry's container execution engine. So it knows how to run a container on a local host and manage the processes inside of it. But intentionally, it's not designed to have visibility into the broader distributed system around it, and that's where Diego comes in. So the Diego system manages these instances of garden running across hundreds or even thousands of container hosts and understands the lifecycle of the containers that it's managing in a Cloud Foundry context as well as trying to decide where best to place those across those hundreds of hosts. So Diego relies on some external dependencies to operate. Principally, it requires a consistent data store to keep track of all this distributed state. So nowadays, that is a SQL database, such as MySQL or Postgres. It also relies on an external system for some amount of component coordination and discovery. So for the past few years, that's been console, and we've been moving away from that and introducing a new component to Diego that we call locket to help deal with some of the lock coordination among those components. And that, too, relies on this consistent data store in the form of a relational database to operate. And then finally, Diego also brings in some additional systems, most notably the SSH access system that you can use to get interactive access to your application containers. OK, so with that overview in mind, I'd like to give you a picture of what a typical Diego deployment inside of Cloud Foundry looks like. So this is something that you might see deployed with, say, CF deployment. If you're using that to create your Cloud Foundry deployments with Bosch. So if you've been looking at any of those deployments, you're probably most familiar with the VMs or instances that we call the cells. And so these are the container hosts within a Diego deployment. So naturally, they're running a copy of Garden to create those containers and to run processes inside of them. But as I mentioned, Garden doesn't know anything about this broader distributed system. And that's where Diego comes in. So on those cells is co-located one of the core Diego components that we call the rep. And this is responsible for registering the presence of that cell in the rest of the system, controlling that local garden and telling it what containers to create and what ones to destroy, and also keeping track of virtual allocations of memory and disk as it represents its remaining capacity to the rest of the system. And then finally, the rep does some amount of asset caching as it downloads things like build packs and droplets to run inside of those containers.  You're not going to end up talking to these reps directly. Instead, you're going to talk to the public API of Diego. And that's presented through the second core component called the BBS. So that presents that public API for clients. But it also understands more of the details of the lifecycle policy for the types of work that Diego runs. So those are principally what we call long-running processes or LRPs and tasks, which are one-off pieces of work. So when clients come in, they tell the BBS API a description of the work that they would like it to run. But when the BBS goes to run it on the cells, it doesn't talk to them directly. Instead, it delegates to the third core Diego component called the auctioneer. And that auctioneer is responsible for communicating with all the cells that are registered and deciding an optimal placement for new units of work. OK, next I mentioned that we are weaning ourselves off of console for some of the component coordination. And so the final core component of Diego that I'll mention is this new locket component. And that presents a lower level API for registering locks and presences to the rest of these components. So the auctioneers can coordinate over having only one of them running at a time so that they're not stepping on each other's toes as they try to distribute work. Finally, since this is a complicated distributed system, the BBS also conducts a periodic convergence assessment on the deployment. So it compares the desired state the client's told it about with the actual state that it thinks is running on those cells and fixes any discrepancies by stopping work that's not supposed to be running anymore or starting new work that has disappeared. OK, so let's now further zoom in on one of these cell VMs to see how that's been evolving over the past year or so, especially in conjunction with new containerization standards that have been emerging in the broader ecosystem. So we'll take a look at this cell VM. And in the beginning, it was simple. There was just the rep and garden linux. And the rep told garden what to do in terms of making containers. But then about a year, a year and a half ago or so, the Open Container Initiative started their first standardization project called OCI Bundles. And these are a lower level description of how to run a containerized process on a host such as Linux. And along with that came a reference implementation called Run C. So the garden team looked at this and said, this is a great opportunity to stop owning a lot of this containerization logic inside of garden linux and instead to start delegating to this community standard for creating containers. So they took garden linux and transformed it into a new component called Guardian that knows how to delegate to Run C. So together, these form garden Run C, which is the replacement for garden linux. And I think if you have an up-to-date Cloud Foundry deployment, you've converged on that as your container engine now. So in parallel with this, the CF persistence team started up and has been working inside of the rep component to teach it how to talk to volume plugins so that containers can connect to distributed file systems such as NFS. And that effort has been becoming standardized in something called the container storage interface. Likewise, there have been networking interfaces emerging in the community, most notably the container network interface. And so the CF container network team has been working with the garden team to take all of the networking complications out of Guardian and instead move them into something external that operates over the CNI networking interface. Finally, the second standard to come out of OCI deals with the layout and format and representation of images, the root file systems that constitute containers. And so a parallel team to garden has been operating called GrudFS that's building tooling and an image plugin to fit into Guardian to manage that. Again, taking more logic out of Guardian into this smaller component that is adhering to these community standards. So the picture of the services running on a Diego cell has gotten vastly more complicated over the past year or so, but it's really reflecting all of these standardized interfaces that are emerging in the container ecosystem and that we're trying to take advantage of. So I know there's a bunch of other talks on this track in particular that are discussing those efforts and where we see that going in the next year or so. So that's very exciting. Another exciting thing is that we're bringing all of this to Windows, too. So the Garden Windows team has been looking at this OCI bundle standard and saying, well, we can make a Windows implementation of that, too. And they've done so. They call it WinC. And even today, you can experimentally run Windows cells on Windows 2016 that will make Windows server containers. So we've got another talk on this track discussing that effort today. So changing gears, I'd like to tell you a little bit about some of the features that Diego has helped enable inside of Cloud Foundry over the past year or so. And one of those is enabling placement for this feature that we call isolation segments. So isolation segments are this capability inside of Cloud Foundry of having a dedicated set of compute and now routing resources going to a particular partition of workload running on the platform. So if you need really strict isolation at the infrastructure level, maybe you want to give application instances more dedicated CPU resources or access to some special device or something like that, then isolation segments are intended to address those placement concerns. So let me give you a small example of how Diego fits into this in terms of deciding placement for those application instances that are in different isolation segments. So one aspect of that is knowing which infrastructure resources are available for placement. And so for Diego, those come in the form of the cells as compute resources. So here, we have initially just a Linux cell running SayGuard and RunC. And when we're setting up an isolation segment, we make new cells, but we tag them with special placement tags, which I'll represent with different colors. So we're now deploying a new Linux cell, but it's got this green tag on it, which means it's going to form a different partition, so it's going to be in a different isolation segment. We can also do this for window cells. Let's add a window cell that's tagged green. And we'll add another window cell that's tagged with a different color, this purple one. OK, so to see how this plays out, we now need some applications that are assigned to those different tags representing their isolation segments. So to start, let's say we have two instances of a Docker image-based app, and it's untagged, so not assigned to a specific isolation segment, which means it goes into the default one. And then for variety, we'll have a Windows app that's tagged with that green tag, and a Ruby BuildPack app that's tagged with this purple tag. So Diego's job from this perspective is to place these application instances where the operating systems and those placement tags both match. So looking at these Docker instances, there's only one cell that they both can land on, namely that original Linux cell that was untagged. So Diego will put both of those instances on that upper left cell. Moving on to the Windows app, there's only one Windows cell that's also tagged green in this deployment, so it's got to place that instance on that one. And then finally, Diego's going to look at this Linux-based Ruby BuildPack application that's tagged purple, but it's going to find that there are no Linux cells that are also tagged purple. So it's not going to be able to place that one. So maybe that's what the deployment operator intended. They didn't want to provide any Linux resources in that particular isolation segment. So it's a runtime error to run that kind of application in a space assigned to that segment. OK, one other feature that I'd like to tell you about that Diego has introduced in the past year or so, and that we're really excited about in terms of enabling more security-focused and stability-focused features on the platform, is something that we call instance identity credentials. So let me tell you how that's set up. Again, let's take a look at one of our typical Diego cells running the rep. So when the rep is making a Cloud Foundry application container, it has a set of identifiers that it knows are associated inherently with that application. So especially in the presence of container networking, each one of those application containers is going to have a unique IP address on the container network. And so that's tied to the identity of that container throughout its lifetime. And then likewise, in a CF context, we know that that application container corresponds to a particular application GUID coming from Cloud Controller. And then Diego even assigns each one of those containers a unique instance GUID that uniquely addresses it throughout the deployment. So before the rep starts any processes inside of that container, it's going to create a new certificate key pair and then supply it inside of the container so that when we start application processes inside of it, they'll have access to that. And they can use it for, say, TLS communication. So let me show you how the contents of that certificate metadata encode this notion of identity on the platform. So if you've ever looked at that certificate metadata, you'll know that one of the most important fields is the subject distinguished name. And that's where we encode these CF-specific identifiers. So that application GUID comes in as an organizational unit. And the instance GUID comes in as the common name in that subject distinguished name. We also provide those identifiers as subject alternative names so that IP address that's uniquely assigned to that container is present as an IP SAN. And the instance GUID is present as a DNS SAN. So one other feature of these certificates is that they're, by default, very short-lived. So let's say that we created this container right at the start of this talk, so 940 UTC. Well, this certificate, by default, is going to be valid only for a day, so only until tomorrow at about this time. So then, of course, each one of these certificates has a unique serial number associated to it that's randomly generated. OK, well, since the certificate expires, we don't want to leave this application hanging high and dry when that day is over. So the other job of the rep is to continually rotate these credentials to generate new ones with a new validity period, and to make sure that they're continually supplied to the application. So right before this certificate expires, the rep is going to make a new one, so it's going to contain the same subject identifiers as the previous certificate. And it's going to have a new validity period that overlaps slightly with the previous one, and then, of course, a new unique serial number associated to that certificate. So then the rep moves that into the container, replacing the previous set of credentials, and then it's, say, up to the application process to reload those or to detect a file system event to know that those certificates have changed. OK, so you'll be hearing in some other talks today about other applications of these credentials. One thing that, if you were at the unconference last night that Shannon Cohen mentioned, is that we're using these to make sure that the go router is always talking to the correct instance, the one that has a registered endpoint for. And then one other application that we have in mind is allowing applications to talk to configuration servers, such as Credhub, to securely retrieve credentials that only they are authorized to retrieve. But I'd like to give you a more intrinsic example about how this can enable better security between microservices communicating directly on the application platform. So to illustrate that, let's have our typical CF deployment and focusing on the routing aspects of this. Here's the go router, which allows ingress for that HTTP traffic to application instances. Let's say that we have a fairly conventional two-tier web application. So we've got, say, a front-end app. Maybe we've got a couple of different versions of it running. And then there's a back-end app. But let's say that that back-end is doing some more secure processing. Maybe it's handling credit card numbers or something like that. And we don't even want it exposed to the public internet as a result. So we can achieve direct connectivity from the front-end applications to the back-end via some of the container networking access policy directives that are now baked into the CFCLI. But because of the sensitive nature of whatever this back-end application is handling, maybe we want to be even stricter about security. And we want the front-end instances to talk to that back-end using mutual TLS, and even for the back-end to be very particular about which applications it authorizes to make requests to it. So to illustrate that, let's put the application GUIDs for these two different copies of the front-end on the screen here. And let's say that the back-end is configured only to accept requests from the new copy of the front-end, so with this A83 application GUID. OK, well, now let's say that we're sending requests for our great website. Let's call it nido.com to the go router. And because we've arranged the routes correctly, those are going to go to the new copy of our front-end app. And in turn, they're going to connect securely using these application instance identity credentials to the back-end. So the front-end is going to see that it's connecting to the back-end that it intended to, something that's presenting a certificate that's signed by that application certificate authority. And the back-end, in turn, is going to get that automatic check through the TLS handshake. But then in its own application logic, it's going to check that the app GUID coming from this client matches the one that it's configured with. So in that case, it does, because the application GUIDs match, and so the request succeeds. Well, let's say that we left an old version of our website running, and it's pointing now to the old copy of that front-end application. But we didn't update the back-end. In fact, we intentionally took that application GUID out of its authorized list. So that front-end application can talk to the back-end. It may handshake just fine, knowing that it's talking to the correct one. But the back-end can refuse it and say, wait, you're not on the list to get in. So that request is going to fail. So I've put some example-going applications that illustrate these interaction patterns up on GitHub in this TLS example apps repository on my GitHub account. So if you're interested in playing around with that, all you need at this point is an up-to-date Cloud Foundry installation running container networking, as well as opting in to generating these instance identity credentials for all of the containers. OK, the last main topic I'd like to tell you about is some additional operator-focused tooling that we've been developing over the past year or so. We call it CFDOT, which stands for the CFDiego operator tool kit. So this is a command line tool intended to interact with the various Diego APIs. So to date, we have commands that address the BBS API endpoints. So if you want to inspect and manage your long-running processes or your tasks, and we've just added support for the API endpoints on the Locket Service. So if you want to dig down and inspect the locks and presences that the rest of the Diego components are coordinating over via the Locket API, you can use CFDOT to do that. Coming up next, we're planning on making a few refinements and improvements to complete and flesh out the API that the Diego cells present through the reps. So we'll be adding corresponding commands to get that information. And at that point, we're well poised to have some higher-level commands that give you more of an overview of what's running in your deployment. So you may ask, why do we need this kind of command line tool to interact with these APIs? One of the reasons is that these APIs tend not to be very friendly to things like Curl. By default, we require all of these APIs to themselves, require mutual TLS to communicate with them. And then once you get past that hurdle of supplying the correct certificates, then you're dealing with protobuf encoded data in more of an RPC calling style. So it's not restful at all. So having something that intermediates that and instead just gives you a friendly stream of JSON objects to pipe into other programs such as JQ is really a benefit to operators. And in fact, we even have a Bosch job inside of Diego release that deploys a compiled CF dot executable and puts it on the path along with JQ. So you can jump on to a VM and just get to work debugging, figuring out what's going on inside of your Cloud Foundry deployment. So I'd like to give you a couple examples of some processing pipelines that you can build just ad hoc using this tool and then JQ to do some processing downstream. So for one example, maybe you just want a quick count of all the number of instances that are in different states running as containers on Diego. So you can do that as follows. CF dot will let you dump out the information about all those instances, which we call actual LRPs. And then you can run it into JQ and do some quick aggregation based on the state that's reflected in all of those records. So here's an example that I pulled from one of the release integration environments, the infamous A1, if you've heard of it. In this case, we've got about 300 running instances, but about 40 that are backed off into this crash state, and three that aren't running anywhere at all, that apparently we couldn't even place for some reason. Maybe they were in the wrong isolation segment, or we ran out of resources because the cells were too small to accommodate the size of the work they needed to run. So here's one more example. Let's say you've got an entry in the routing table for the go router, and you want to track down the application GUID and index that that endpoint corresponds to. Well, that information, too, is reflected in the actual LRPs presented in the BBS API. So again, you can dump out those and then pick through them to find the host IP of that cell VM and the port that's registered for that application in the routing table. So again, here's an example from that A1 environment with a particular IP and port. And it allows us to narrow down to a specific application GUID and even the index of that instance. So if we then need to restart that instance or try to find something about it in Cloud Controller, we now have that core identifier from the CF context. OK, and then finally, I'd like to mention a few things that we're considering for Next Steps in Diego over the next year or so. One thing that we're very committed to is having the appropriate support for zero downtime or rolling updates inside the platform. So we did some initial exploration of that work over the past few months. And we've been discussing with the CAPI team how best to implement that so that we can give the correct routing availability guarantees. So figuring out how we're going to coordinate as necessary with the routing tier to bring this out is really the next challenge to bringing the solution to all of you. We know that a lot of the blue-green and rolling deployment scripts that everyone is using are doing this kind of routing gate checks to make sure that they're not destroying the old copy of their application before the new one is up. And so we'd like to make the same kinds of guarantees as part of the platform. We'd also like to better understand how we can place large application instances while requiring operators to maintain less headroom in their deployments. So that may come in the form of doing things like active rebalancing, moving application instances around, or dynamic over-commit on cells. And again, if this involves moving instances around dynamically on the platform, we'd really like to make sure that we're maintaining the kinds of availability guarantees for those application instances, both in terms of their intrinsic health, whether or not they're running and whether or not they're routable. Next, we've definitely seen some cases where certain individual cells or maybe the entire cluster has gotten kind of overwhelmed with the amount of work that it has to do. So we've put in some crude controls to gate those if you know specific things about your environment or if you're reacting to past failures. But we'd like to see what we can do to make that kind of back-off more automatic based on more information coming from the cells. As you may have noticed, it's been almost a year since we cut Diego version 1. And so we'd like to cut another major version pretty soon in part to simplify some of the deprecated properties and API endpoints that we've had over the past year and also to reduce some of the configuration choice and complexity. So things that we're considering requiring for V2 are making components always communicate with each other over TLS. That's always been the recommended configuration for any production deployment. But it's been an option for development. We found that with all of the new tooling enhancements with the Bosch CLI and CF deployment and things like Credhub, that generating and managing that zoo of TLS credentials is much less burdensome at this point. And so it's already the default for, say, a Bosch light deployment that you're deploying with CF deployment. And so just requiring it for all the components seems like a sensible measure at this point. And then we'd also like to continue weaning ourselves off of console. So we're considering making a locket a required component running in V2 so that the other components wouldn't operate without it deployed. And then, of course, we're always committed to any stability and security improvements we can be making. And we really value all of your feedback from the community in terms of identifying those. So we can't observe every failure about the system firsthand. And so if you encounter those, please let us know about them. So for the next few minutes, I'd be happy to take some questions. And otherwise, thank you so much. I'm emolmepivotal.io, emolme on GitHub, and in the CF open source Slack. So please drop by the Diego channel and ask us questions. Yeah. Regarding instance identity certificates, have you put more thought into how you might delegate signing to an external certificate authority if the enterprise you're working with has got such a thing? So do you mean if you as an operator want to supply another CA to generate those? Yeah, exactly, yeah. Yeah. So theoretically, that's configurable inside of the Corbash release. So each one of the cells just needs an intermediate CA and corresponding key to issue those certificates. So if you can generate that from some parent certificate authority, then you can just plug it into the Diego deployment. And it'll generate things associated to that chain. And then we also have various capabilities of installing root CA's inside of the containers themselves. If you're using build pack apps, it's very easy to do that just to bake it into the CF Linux FS2 root FS. But then we can provide them on the side for other containers where we maybe don't know so much about the root file system. Yeah, Marco. So in your mutual TLS example, you were using app GUIDs to validate that the request was coming from the right application. I guess that's in most cases not what I actually want to be using, right? Probably I want to allow traffic from all applications coming from a certain space or a certain org. I maybe want to use the name of the application so I don't have to update every time the GUID changes and so on. Is there anything thought about or planned already in that direction? Yeah, so it's definitely something we've considered. I think it'd be very easy to provide those Oregon space GUIDs as additional organizational unit identifiers in the subject. So that's probably the next logical step to take there to enable less granular and less fiddly authorization primitives for those certificates. Dealing with application names is a little bit more difficult because those are mutable in the CC API. So you could end up changing it, and then we'd have to propagate that update down to all the certificates. But because we are able to generate them dynamically and to replace them in all the containers, that wouldn't be infeasible. We wouldn't have to restart all the containers when their name changes. So I think that's something that we could do if there is sufficient interest in the future. Eric, can you say something about you said, in looking forward, you're trying to, well, exclude stressed cells from placing new containers onto that, like, any word about how you intend to do that? Like, how do you measure when a cell is stressed? Like, I don't know. Just lay out what you guys are thinking around that. Yeah, so that's a good question. How would we measure that degree of stress on the cells? So some things we can do are looking at failure rates for the other dependencies that, say, the rep has, like talking to Garden, making containers, deleting containers. Since the rep is also involved in downloading various assets, if that's going slowly or timing out, understanding those failures and folding them into something more like a circuit breaker system for the cells. So the hard part of that is tuning that kind of information correctly so that you're not backing off of cells too quickly when they're still fully capable of doing work. They're just going a little bit slower than you might expect. All right, well, I think that's it for questions. Thanks so much for attending, and enjoy the rest of the talks. Thank you.


I hope that's like the candy. It looks like we are live. Hello, everyone. Thank you for tuning in to hear from a great group of executives on our panel today. In case you are wondering if you are in the right room, we will be discussing why open source contribution is essential for business.  I'm incredibly excited to see people here from all over the world. This DrupalCon is truly global. My name is Emily White and I run marketing for phase two. Phase two has really been entrenched in this community for a long time building digital products on open technology for almost 20 years now. So you can imagine how honored I am to host these leaders today. We all understand that contribution is a key to accelerating Drupal and other open source projects and millions of sites use Drupal, but yet only hundreds of them actually actively contribute. So the reason why we brought folks here today, this panel is really designed to help us understand the common barriers to contribution, why a contribution culture is good for business and inspire us with some new ideas to walk away with. So if you have any questions for this panel, please add them in the chat. We'll be answering as many as we can. We don't get to your question before time is up today. We will follow up and reach out individually. So I'd love to introduce you to our esteemed panel we have today. So this hour you get to learn from Derek Gilmore from Pegasystems, Deb Bryant from Red Hat, Nikhil Dashbande from the state of Georgia and Guy Martin from Oasis. So panelists, thank you for joining us. I think Deborah will probably hop back on. She's having some thunderstorms. So hopefully she gets back to it soon. Panelists, I'm going to ask each of you to give us a quick intro for the group. Tell us a little bit about how your organization contributes to open source projects. So I'm going to pick somebody. Let's go ahead and start with Guy. No, why don't you kick us off? Awesome. Thanks, Emily. Hi, everybody. My name is Guy Martin. I'm the executive director for Oasis Open, which is a standards development organization that also has a strong open source component. I've been in the open source world for quite a while. I spent some time helping Red Hat do strategic open source consulting, and I've started several open source program offices at places like Samsung and most recently at Autodesk. And I think where we've done a lot of contributions in those organizations that I've been a part of, but I think what's been most interesting to me in the five months I've been at Oasis is that Oasis doesn't contribute directly to open source, but we're an enabler of open source contributions. We have a thing called our Open Projects Initiative, which is really about bringing open source projects in and hosting them in a traditional way that a foundation would, but also giving them an easy path to disure a standards recognition. Because I think what's interesting is that it's kind of a little bit of a back to the future thing. Every thing that we're relying on here as the panel to actually be part of this conference is based on open standards that also have open reference implementation. So things like HTTP, et cetera. So I'm really excited to be here. And so I think our friend, Derek, is going to also give his introduction and kick us off. Thanks, Guy. I'm Derek Gilmore, Senior Director of Web Architecture for PEGA Systems in Cambridge, Massachusetts. I lead the digital team at PEGA. Been there for five years. We have a team of about 30 people globally. We have people in Cambridge, Poland and over in India. When I started five years ago, we had two people on the team. We've built it out to a fairly large team. We use Drupal for all of our public facing websites and also for internal portal and also sales enablement websites. We have, I think about nine different properties that are using Drupal in the organization.  In terms of our contributions to open source, a lot of the modules that we use, we get involved in maintaining patches and bugs, you know, fixing bugs and things like that. Our lead developers are very active in that. We also hold events at our company. PEGA has facilities for doing training. So we often do meetups at PEGA in Cambridge and we also do them in Krakow, Poland. We also sponsored Drupal camp in Poland. We've done that twice and put a big presence there. We've also done DrupalCon Europe. We've done birds of a feather sessions. I'm also pushing all of my developers to have an active presence on Drupal.org and pushing them to speak at conferences as part of their own career development path. We push all of our developers. Our head of HR came in as really pushing personal branding in terms of helping with recruiting. So when you talk about how do you help an organization, how does contributing to open source, people looking at recruiting talent and how do you find that talent? Well, you have to have presence within the Drupal community. When I first started with the company, our recruiters would reach out trying to find talent and they would say, you know, what's PEGA? How does it all fit together?  I saw your lead developers speak about, you know, caching at DrupalCon Seattle. I saw this person speak about design systems. So the conversations are much more productive when we talk about having people come into organization. So finding Drupal talent or any tech talent and highly skilled tech talent these days is very, very difficult. So having a presence within the community is critical around that.  So that becomes very important. And then we also want to get in, the big thing on the net horizon is try to helping people learn or new people get into Drupal. And how do we make that happen? So it's a little long-winded introduction, but that's kind of where we're at with Drupal and trying to help the community. That's awesome. Yeah, Nikhil, what's happening at the State of Georgia? Thanks, Emily. I'm Nikhil Deshpande. I'm the Chief Digital Officer with the State of Georgia. And I head the Office of Digital Services within the Georgia Technology Authority. So essentially our charter is to be the consultant slash partner for state agencies and elected officials to make sure service delivery happens in an effective efficient way. I guess we have been using Drupal for about the last 10 years. In 2010, we decided to move to Drupal. So we moved all the state websites around that time to Drupal. And ever since we've been growing our footprint and at this point we are happy to say within the state, Drupal is the official web hosting platform and content management system across agencies. And we have been an enabler for contributions for sure because our model is more around outsourced where we work with development partners and hosting partners. We have a very slim staff of mostly strategists in-house but we work with partners and we have a very strong inclination of partnering with partners who have a strong presence and history in contribution. And we always have had that charter in the last as many years as we have a Drupal presence. And for that we have certain guardrails in even partner selection in how we treat our projects and how we enable our development efforts to make sure that we always are looking at a contribution mindset. And although we ourselves as a state of Georgia though we have an account on D.do our kind of branded contribution might be very slim but through our vendor partners we are very happy to kind of be making sure that a lot of the stuff that has been developed for us for on our projects is developed with the mindset of being contributed by. It's great. Thanks, Nikhil. Well, I think Deborah started to hop on for a second and then fell back off but I'm sure she will hop in when she can. I do wanna address sort of what I've heard from the three of you. Obviously the reason you were chosen for this panel is cause you have sort of successful contribution type cultures from different angles, right? What have you seen out in the market as being barriers to this kind of culture? What have you seen as either within your own organizations as you were developing this culture or in the other organizations? Yeah, I mean, I think what I've seen in previous cultures and I won't mention names some may be able to figure out which ones I'm talking about based on my background but I think what I've seen sometimes is one of two barriers, either the legal team not being comfortable with it or the business leaders not being comfortable with it. And generally what I found is in partnering with legal teams, they will defer a lot to the business leaders. And so the way I've sort of tried to attack this and finding a way to make this happen is to go to the business leaders and have a very frank discussion about the fact that if you don't contribute, that's fine but there are consequences to not contributing. So I have a specific example from a previous organization that I was in that had G Streamer. We use G Streamer in our products and the team decided they were gonna fork G Streamer and have an internal fork that went on for a long, long time through multiple product lines. And then they ended up finally kind of coming to their senses a little bit and saying, hey, we fixed a couple of bugs we wanna contribute those back. Well, the problem at that point was that it was so far down the path the main line was so far down the path that those bugs had been fixed A and B there were certain like P1, 7, 1 bug fixes that needed to be brought back in. So at that point, you've now spent engineering resources this is what I told business leaders you've spent engineering resources and you're gonna have to spend more to bring the main line back in. So I said as an example, that's a great example of not doing it that way because you're actually gonna pay twice. So I said, figure out how you pay once in the sense of contributing back things that are non-differentiated IP and it's really important I think for business leaders to understand what is differentiated intellectual property and what is not differentiated intellectual property. So that's how I've sort of tried to address that with business leaders because generally the legal teams will follow the lead of the business leaders. Right, and I would add to that is where you have to be careful about how much customization. I agree 100% what Guy was saying you have to be careful that you've generalized your contributions because every company or organization has so many unique requirements they start making the spoke customizations or extensions or whatever they're doing to it and having to say, all right, we have to refactor this to actually make it applicable in a more generic sense. So that's a big challenge. And it's, you know, you're saying, okay, we have to make this generic enough so other people can use and you have business people pushing you to say, we have a problem to solve here. We don't care about anybody else. So that's realizing that there's a win by doing that. Great. The only thing I can add to that is just like, there is definitely this culture of alignment of having that mindset of contribution where, you know, people who are aligned with that you don't have to teach them to look at the forest for the trees. They know the big picture and the benefits. And thankfully within our organization I did not have to go and convince anyone about the importance of contribution largely because it's a lot of it falls within, you know, kind of like my domain, my purview. We were able to pretty much make that call by saying, you know, we choose to work with organizations that prioritize contributions. And then accordingly, you know, that just fosters the culture for like anything that we do after that. But, you know, truly kind of, if I were to equate it to something I would just be like, you know, the difference between planning sunflowers versus oaks is eventually it will grow. Eventually it will truly kind of benefit everyone. But being one of the first states to jump on the Drupal wagon and also one of the first states to truly kind of embrace open source for its enterprise platform. I have seen a lot of resistance only because I guess there was a lot of misinformation from some of the executives where having a dialogue, having a more, you know, kind of educational conversation truly helped in building that bridge. It's a good point. I would add that I think business people often have a hard time getting their head wrapped around the fact that you have to be involved with the community to truly understand some of the technical aspects of Drupal that you need to send your developers every year. I have to put, throw down the gauntlet of saying, you know what, I need a lot of people to go to DrupalCon or go to this event. And it's really not just to go there as a boot dog, it's actually to learn and train the people. So when they come back, they can hear that, oh, by the way, React is coming. We need to rethink our front end. And, you know, it's a real thing now. So that, you know, fortunately I've been very successful with that, but I have a very, very large team now and I can't send everybody to DrupalCon now because the team's so big, but they're getting people to understand the value of, you know, the training and learning and that that's how you, you know, are more efficient as an organization. So it's, in addition to contributions, you have to support your developers' training program and say, you know what, you need to go to DrupalCon to make sure you're up to speed in the latest and greatest with Drupal and using it properly and efficiently, I suppose. Because if you take a straight PHP person and they start coding and they're not trained, you can have problems. We all know that. So. For sure. Yeah. Guy, you mentioned like a couple of instances where there were barriers to entry. Have you experienced times where you've had to convince business leaders of the benefits of contributing to open source? And if you have, like, have there been specific key points that have really clicked in a business leaders' mindset to say, oh, this makes sense for my organization or either of you actually also, Derek or Nikhil, if you run into this, I've always been interested to see, like, what is the tipping point to say, I got it? You know, we talk about operational efficiencies and that sort of thing, but what is it that makes it click? Yeah, it's a great question, Emily. And also, I know there was a question that came in that actually sort of addresses this as well, which is that building your business value or the business needs you have into a piece of open source is really, really important, as opposed to necessarily waiting for the community to do that for you. So generally, that's one of the things, and I can use a specific example that is public, which is Samsung, which is when we started the Samsung open source group, the whole reason behind that, that whole open source program office was to make sure that Samsung had a seat at the table for really important projects that we were using at the time for products. And so I think that's a great way to address that with a business leader to say, hey, this open source project that we're relying on for our product, wouldn't you rather have a seat at the table around what the strategy in the future direction of that project is? The only way to do that, you can't buy your way into that. The only way to do that is to make good sustained contributions. So that investment from a business perspective makes a lot of sense because you not only help build something that, yes, you're in some cases enabling your competitors, but your competitors are also enabling you with the contributions they make. I think that's also one of the things that sometimes business leaders get their head sort of wrapped around and have an issue with this. Oh, I'm enabling my competitors. Well, yes, but your competitors are also enabling you. And if you're making those contributions, you as an organization have a seat at the table in terms of what the strategy for that open source project is going forward. Yep, so you can kind of direct the sales, right? That's nice. That's a good point. Derek, Minkiel, anything to add? I noticed a question in the chat someone was asking about how do we educate lecturers in higher education and teach students about the benefits of open source and contribution. It's not just free software. I think it's important for people to understand that open source software is not really free at the end of the day, the software in itself is free, but you need to have people who can help you maintain support, set it up. And there's a really good career opportunity for people to learn about open source technology there. So that's something that people understand that the potential salaries and opportunities, I mean, Drupal has changed my life financially. It's been a very worthwhile investment for the past 10 years. I'm glad I made the switch. And there's even, whether you're in Drupal or any type of enterprise or large scale content management system, there are fantastic career opportunities there. So I think that if people are, educators are trying to talk to students about investing time and learning about this technology, they need to point out, go on to Indeed and start looking at some of the salaries and career opportunities within organizations. There's, it's not just the coders. There's a whole group and ecosystem that's involved with touching these content management systems that it's, it runs a spectrum from designers to UX people to content editors to there's all kinds of job opportunities within this space. Derek, talk to us a little bit more about that about contributing past code, beyond the code. Are there other ways that people can get involved in this who are not developers? Yes, I mean, it's from writing documentation. I mean, I was involved and we had a meetup and we were going through how to set up an environment on Drupal 8 and I've done it myself many times and I was pulling my hair out and Benji, this guy Benji from Isavera and I sat down and actually rewrote some of the install instructions right there on the spot and said, we need to clean this up. So, and you don't have to be a technical user. You just have to be someone who can follow instructions and say, you know what, they're missing a step. They left this out. Let's write that down and put this on there. We'll make things easier for people and there's testing, there's documentation, there's UX improvements. There's also, if you run events at an organization, an onsite event, there are people who have to coordinate all of those activities. You just don't say, hey, come on over and show up from publicizing the event to getting food to preserving the space. It's just all the logistics to make that happen. These things don't magically happen. So all different types of people from the team can be involved and they don't have to be hardcore Drupal developers to be extremely helpful. One of my dev managers in Poland, I mean, it's about making phone calls and reaching out to other agency people and cooperating and say, hey, you know what? We should be teaming up on this and working collaboratively on things. So it's those kinds of things that make it happen. Great points. I can just add to that where, I think supporters of open source need to be more vocal than they have been. And I know all of us do our parts, but at one point I was not an open source native and I had to learn about it. And while we were in that decision chair of to go open source or not, the amount of, I guess, lobbying that happened from proprietary software and CMSs, there's no comparison because there was really, unless someone is personally invested in the concept of open source, it's very easy to get swayed by something that is just because end of the day, open source will cost you something, right? I mean, there's this whole free thing to what Derek was saying earlier is really not true. Like the word free is very heavily misinterpreted. It's not free pizza. It's free speech, right? So it's like the value is that we can actually make a difference when we use open source solutions. And yes, there is definitely a certain boundary where you can customize it to the point of no return or you could align with the community and see where everyone is going. But I guess the key here is just to kind of build that conversation because what I have seen, and of course, like 10 years ago, me being one of the state officials trying to pitch the idea that, hey, state of Georgia, let's go open source. Today it might be a lot easier sell than what it was back then, but it happened and then other states followed quite a bit. I mean, there were like a few states who were implementing open source for certain parts of their organizations, but as statewide open source solution. I think that really kind of needed that conversation to happen between stakeholders and between other organizations. And I think also having this, for the lack of any other term, this evangelizing community of this is the value of it without getting technical or without counting beans about it, the value of it, the big picture. I think that is also very important because end of the day, we are all in this together. And as what we can see, even though we are not using the exact same implementation of Drupal, but then every organization, we are still using the same product. So anything that we invest in automatically, other agencies benefit from it. I would just add there that those that may know Denise Cooper, who is very well known in the open source world, she has a great concept here around businesses that this is enlightened self-interest. This is not altruism, even though to your point, Nikhil, there's a lot of great things that happen from an altruistic standpoint, but this is enlightened self-interest. Businesses have a value that they gain from using these open source projects. And then contributing back, they also have a value to gain, part of which is also potentially, again, driving the project in a direction that's beneficial for them, as long as it's also beneficial for the rest of the community. So when I talk to business leaders, I actually use Denise's commentary about enlightened self-interest quite a bit because I think it gets past that whole, oh, we're just doing this for nice PR reasons or to be good people. We are, right? And there's obviously value you gain from that, but it is really an enlightened self-interest culture. Right, and I would also add that vendors or a lot of people are thinking about they don't want to have vendor lock when buying a commercial product. So if you look at the commercial CMS solutions out there, you're tied to their feature roadmaps when you want enhancements. If you want a very unusual or highly customized thing, you're gonna have to hire consultants within that proprietary system and they're as expensive, if not as more expensive than the open source consultants, usually more expensive because it's more of a rarity. I won't name names, but that's something that's on the table that we'll be able to think about that most of the time there is customization whether you're using open source or proprietary. And then you've got a situation where, all right, well, how much is this customization gonna cost? And then am I going to be locked in? I mean, an open source or maybe you may be able to switch to another open source solution if needed. So that's something that people need to take into consideration and why open source can be more beneficial in being going that path. Absolutely. Nikhil, you mentioned that people need to be a little bit more vocal about their involvement and their support in the community. You mentioned at the beginning when you were introducing yourself about the value of partners and vendors also being involved in this with you. So how are you sort of tying that contribution culture to partners and vendors that you work with? One of the things that I guess through the years we saw that open source being a culture obviously impacts how development partners, vendors also operate. And that defines the individual company cultures. And we have been particularly in our case very fortunate to be always partnered with someone that we enjoyed working with and in the most recent procurement that we had to go through, we made it a very clear point to only engage with vendors who had a clear track record of contributions. And when we stated that literally in our RFP as documented requirement, of course my phone would not stop ringing with request to take that off because and then also started seeing like some more contributions by some vendors, of course I'm not gonna name names were like before submission two weeks prior to there was like this token contribution and that was just glorified in the response. So the key is to yes, I mean, work with vendors that have that culture but also as a business have the appetite and plan for paying for that when your project is in development because requesting someone to have a contribution first development culture and then not planning for it when they actually do it on your project is not fair. So accept the fact that sometimes timelines may shift because they have to approach a certain development methodology in a way that it's kind of contribution focused. So that is also from a business perspective we kind of have to be ready for that. And that is something that we have done and we had some very strict criteria for contributions. We had strict criteria for development vendors whose primary revenues were from Drupal projects. So it kind of automatically disqualified a lot of the other prospects for us and truly got in the room, the real serious players within the space and of course with contributions to, I guess, Derek's point earlier. Yes, there was a way for us to kind of read what they have done from contributions but they also named their people who have almost kind of like a for the lack of any other term I'm gonna call a rap sheet of what the contributions have been in the past. So this is great. I mean, just sometimes looking at the names we can identify some of these rock star developers. So I think that really helps because end of the day is truly this open culture that we know who's playing in it, we know what they have done with it. So that just opens future possibilities for them. I would add another dimension to think about is that people who are contributing to the open source code have to clean up their acts in terms of the quality of the code that they write. So, Mikhail putting on there that I want a firm that contributes immediately is setting a bar that says, I want people who write high quality code because a lot of eyes looking on the code and there are a lot of people in the community who are looking at it and saying, no, no, no, this is not acceptable. You need to clean this up. And that is a big deal.  But the question is, is it clean? Is it scalable? Is it secure? That's where the benefit of contributing to the community other people are looking at your code. There's security standards. That's a huge deal. That's in, if you want, quality code. That's why a lot of, you know, government organizations are running their sites on open source because there's a lot of eyes looking at it. And if a lot of eyes look at it, they're saying, oh, we got a hole in this. Let's plug that. Let's clean up the security on that. It's something that's quality is, you know, it's almost counterintuitive.  All these people are just messing with it. And it's like, no, there's actually a process for keeping the quality up. And that's a huge benefit to people who take the code to use it and also for contributing to it because there's standards around that. All right. Though the one thing I do want to point out, Derek, by the way, that's a fantastic point, but it's also really important that we talk, I think, maybe about sustainability in open source. And I don't know, Emily, it's probably I'm hijacking. Maybe it wasn't in your original questions. But to your point, Derek, I think that we really do need that sustainability and it's important that you have healthy communities around that, right? I mean, there's a reason why the Linux Foundation started at one point, the community core infrastructure initiative because you have certain open source projects that we all rely on, NTP, open SSL, that kind of had the opposite problem of what you're talking about, Derek, that they're being relied on, but there aren't a lot of eyes and there aren't a lot of contributions to them. And so that's why I think it's really, really important that we have healthy ecosystems and healthy communities. I mean, it's really kind of also kind of the flip side of when you're talking with business leaders around consuming open source. One of the things I've always told business leaders is that you should be thinking about not just the technical fitness of that code, right? Your engineers may say, hey, this is a great piece of code, but if you're auditing and looking at consuming that, if that open source community isn't healthy, if that's abandoned, pulling that in is just asking for trouble later on. So I think making sure that open source has a sustainable path and has folks that are making those contributions and those investments is important. Yeah, well, that's, I mean, you look at Drupal moving to GitHub and to Git and then you can look at a project history, you can look at the recent contributions, you can start seeing the health of things. So anything, that's what everybody's looking at these days. They're like, all right, when was the last contribution made to this module or this piece here? Who is the active contributor? So all of that comes in a play that it's this self-feeding thing and it can die on the vine too, just like you said, you find out something nobody's been working on that for two years and you have to decide, do I want to use this or not use this? Yep, it's something to consider. Sure, this is great conversation. I've already learned a lot. I think we can also take some questions from the crowd. We have quite a few in the queue here. We'll get to as many as we can. Thank you, gentlemen for sticking with us. We have a question from Samina Pereira and they said, how can open source software and applications compete with well-established market pioneers of paid software? And what strategy should a budding open source organization follow for success? That's a good one. That's a lot. That's a lot in there. That's a lot. Samina, complex question. You have to look at somebody, you have to talk to somebody and say, what's your end game? What's your, in terms of your budget, open source? I mean, you can look at the stuff that's going on with Amazon these days and all the tooling that they're putting out, they're putting out, I mean, you can literally start a company that has tens of millions of dollars with the technology and your fingertips with no investment. And that's all because they've open sourced or released all, and that's open source, put it out there for free essentially. So the question becomes what problems are you trying to solve? What's your business plan? What's the strategy? What market are you going after? There's so many opportunities out there with using whether it's open source or combining a service at AWS, for example, they have search right now with very search in the cloud that can be weaved into Drupal and you can do all sorts of crazy things with that and there's money to be made for people who can help set that up because it's not you plug it in but then how do you make it work right? How do you optimize that opportunity? I mean, this is the best time to be in the software business in my opinion that the tools that are at your disposal and platforms and services at your disposal if you have the talent, knowledge and ability and interest, it's disguised the limit. I mean, it's a great time. I mean, if you look back even 10 years ago it's we are so far ahead of the game today and look at like machine learning as a service. So let's plug that in and put AI on our search and make it better. I mean, that would have been a pipe dream 10 years ago. So there's a lot of stuff around that. I don't know, that's anyone else. I could go on for a while on this one. Anything else? We'll move on to the next question. That's good, great. All right, how can we as business leaders address the diversity, equity and inclusion concern that not everyone has free time to contribute? I asked the thoughts, but I've been talking a lot. So, Nikhil, do you want to... I'll pick on somebody, put somebody on the spot. No, I mean, definitely that is true and that is something that we have to consider that not everyone is fortunate with extra time. But I guess that's kind of like why it is important for the business leaders to have, create kind of like a setting in which you working on a project really could be considered something that you could be contributing to. It should not be like a weekend thing. It should not be something that you do your day job and then, right, because that truly puts a lot more load on someone. And if you're not really equipped with your time and you are bandwidth to do that, it is not fair. So I think that is truly something that organizations and business leaders across organizations really need to consider in their investment in open source contributions. Right, I mean, and to that point, Nikhil, I think it's really important, I'll use Samsung again as an example, because one of the goals of starting the Samsung open source group was to have people who are working on core open source projects on behalf of Samsung to get, again, that leadership. But that was part of their job, right? Essentially, that was their job. And even those people who didn't make contributions from our team, and as we mentored other developers inside of Samsung, it was figuring out how the open source project that we wanted them to contribute to aligned with their day job. Because then, at least you may still have issues with bandwidth and time, but at least if it's aligned, it's a lot easier sell to their boss and to the business leaders of why they should be making those contributions. Yeah, I mean, I would add it's up to the leaders within organizations as we said, create space. Open up people's calendars to allow them to invest time into this weekly, monthly, going to events. I can tell you that some of my team that's over in Poland, having them come to Amsterdam, some of them have never left Poland, and to drop into Amsterdam and to go to a DrupalCon and see the diversity there, some of it was mind blowing for some of these people there culturally and to see how diverse the Drupal community was. So I think encouraging people who within your teams to go to these different events, to giving them, to supporting that paying for them to attend. And I would also add that is an individual investing your own money if you can afford it. In these things, it's almost like your own education that this will pay back in your lifetime and in many ways over because you will increase your network of people you know within the community, which is the people who do invest their own personal time and effort into it on the weekends, in which I know most people don't have specific events. It comes back to them because you end up having a network of people and friends that turn off and turn into career opportunities because when we ask people in our Drupal team who to hire, the first thing I do is ask people in our team and they're reaching out to the people that they've worked with. So there's a whole social network around that. So getting involved and putting these people who do have the time and money are doing stuff even on their weekends, on their own personal time, just to invest that personal investment. But we do need, and I know we're also the Drupal organization trying to give people can afford it scholarships or whatever. I know that that's probably out there. There's some of that out there to help them out. Definitely. I think we have time for one more question and then we'll have to start to wrap it up. And I'm gonna take my personal favorite, which is how do I get started in either fostering a contribution culture or even as an individual in contributing to open source projects and the community? Any advice? I'm holding that. Guy, what are you, somebody goes to the front. I'm holding that. I'm like, not everyone at once. It's like, I've been talking. I'm trying to give everyone else a chance. I think a couple of things. One, finding a project that you're passionate about, hopefully one that aligns with what you're doing in your day job. And then just figuring out how you sort of make that business case to your management team about why that contribution is important. I've seen in organizations, people do side projects that are of interest to them. It's generally harder to get those through, I think, as far as doing those within the context of the business. And there's usually some conflict of interest, even if they're doing it on the side. So finding a project that you're passionate about that aligns with what your team is doing and then making that business case around that this isn't an extracurricular activity, that this is part of your job, and then this is an important part of your job. I would add, you need to look at putting up a dupal.org profile. If you haven't done that, you need to do that today. Step one. You need to put your organization's profile on there and tell your organization they need to have a profile. And then they need to contribute some money a little bit. Get the name out there, get the logo out there. And then, like you said, find a project that you want to work on. And also look at other people's profiles. Come to dupal, hold a dupal event locally. If you don't want, attend a dupal event locally. See what they're like. If you don't have one in your area, once we start all meeting, you go virtual right now, but then once we start COVID ends, we at some point, think about having one in your area. Miquel, parting words? Yeah, I mean, I'm kind of like, I'm looking at it from my developers, who have basically contributed. And this is someone who kind of sits along with us and our teams. And I think it's more about just having that conversation and just speaking to your supervisors or whoever it is that would assign you that bandwidth. And just always keep talking about the importance of it. And I mean, eventually they'll get it. I mean, I hope they do because that's really what I have seen. And it's a sense of accomplishment. And this is, of course, on top of everything that Guy and Derek have said, like you identify the project that you wanna support, you start contributing.  And then truly that becomes like your family. And you can help, but then start building your network around you who can help you in that project. So it's a really good way of growing yourself as a developer. That is wonderful. So it's awesome to be walking away from this with three ways to start. Fill out your Drupal.org profile, get your company to do it. Find projects that align with your organization's vision and mission and goals and evangelize. Be vocal about what you're learning with others. So those are all three great points. And thank you three so much for being on this panel today. We learned a lot from you. So hope you all have a wonderful rest of your DrupalCon. Thank you audience who's been here. Hope you all learned as well. If you have any other questions, we will address them individually. Go ahead and put them in the chat or reach out to us here online. Thanks y'all. Thanks everybody. Have a great conference. Thank you. Have a great conference.