I will go ahead and get started. I think it was the start time, 235, Missouri. I'm a little late, it's nothing unusual.  I didn't actually get a chance to read it, but I think you've seen it like five times now. So topic here, eyes on target continuously fielding mission-driven value. I'm Keith Strini. I'm the pivotal federal practice lead. So what that means is PCFS, which is our solutions team, is on the delivery side. So we do a lot of the implementation of custom solutions for our customers in the field. And in this particular case, it's a little bit different in the federal government, specifically in the national security sector. I happen to actually come from this background, so I kind of got the designee appointee by Pivotal to be this federal practice lead. And what it was, is we really saw this market of vertical sort of opening up and the need to help do some transformation and not having that exact understanding of that context. So when we started this, this is a real good quote. Before I came to Pivotal, I actually supported Radham Lewis, and he had this quote about speeding up the delivery of innovation while maintaining technological superiority in order to maximize the warfighter's advantage. And it was something that it was the first time I had seen it. This was probably three or four years ago. And it was the first time I had seen it sort of caveated this way, in which it made total sense about how you switch your mindset from a revenue generated like we do in the commercial sector into what really matters in the defense sector. And so I've sort of carried around this quote everywhere I go. It's a nice way to kind of summarize exactly what we're doing here in this particular vertical. Understanding actual security, it's a difficult subject. I know a lot of you, how many are actually in the actual field itself in supporting DOD?  Yeah, so you guys all understand it's not easy. There's a lot of things that we do that's fundamentally different by necessity and some things that aren't by necessity that are just a culture that have built up over time. And so when we came to this at Pivotal, we really wanted to make a conscious effort to translate the shared values that we had on the commercial side into things that aligned with the culture that we saw in the DOD and the IC. So we do things with the intelligence community and the Department of Defense and other civilian sectors as well. But this talk is primarily focused on the challenges that we have in the defense sector. Well, we wanted to do this with a sort of a stubborn intent not to sacrifice those things that we enable our commercial customers to do, but while recognizing there were certain gates that we needed to meet. ATOs, security is different, air gap environments are different. So we knew that going into this that there were gonna be some things that we needed to shift on, but we wanted to make sure we maintain who we were in this space and make sure that our customer had that same sort of shared vision on these different pieces. So the primary goal here was the development and implementation of sort of how could we do continuous fielding and how could we do it well? We knew there were gonna be bumps in the beginning, but over time, how would we improve to make this a very efficient process that matched our speed in the commercial sector? So why go faster? Well, there's different threat domains all over the world and they're all evolving at different speeds. And when you realize that and you think about the different domains that we fight or defend and you have cyber, electronic warfare, traditional means, these type of things, all of those things have very different, different countries have different evolutions, countries that can't afford full-blown aircraft carriers and air force, full air forces, they're investing heavily in cyber. So it's still an attack factor. We still have to be aware of it and we still have to develop capabilities to defend against it. And then you have the traditional, like you saw earlier in the talks about projecting air power and all that goes into those different pieces. And so because of these different speeds, we struggle to stay up with them. I mean, it's basically like fighting on multiple fronts and we all know that's a bad idea strategically. So when you couple that with the speed which Captain Kruger had talked about earlier about how fast things are actually getting to the field and I think Tori had said something like 11 to 13 years, right?  And so we're opening ourselves up to a lot of these different attacks unless we decide as a group how we wanna go faster without cutting corners. And so that's sort of what I wanna talk about today which is what have we done so far in this space and what do we still need to do but then also recognizing that this model is applicable to any of the Department of Defense services, right? This is not specific to Air Force although there are things that were specific, Intel we could do certain things, Air Force we could do certain things and that will change. Navy's got over the horizon challenges, things like that. But we have plans for doing this type of stuff. It's more about communicating the possibility that this really does apply to the defense sector rather than just saying this is a commercial thing and it would never work in our particular vertical. But really this is not about revenue generation. I mean we all recognize that, especially folks in this room coming from this field. It's about how to, in the national security space is about how fast can you deliver a compliant solution to satisfy a mission need. And in the Department of Defense you have a concept called mission effectiveness. And what that means is, is whether or not a mission got executed. And just to give you an example of something concrete is like the tanker planner situation. Tanker planner, bottom line tanker planner was mission effective. It was refueling all the planes that it needed to do. It didn't mean that each actual individual measure was effective. So where we found savings and were able to save all that fuel cost that was an efficiency, a measure of efficiency that could be improved. But at the end of the day they were still fueling all the tankers. And so the idea here is how do you hone in on what those, you know, maintaining mission effectiveness as the bottom line. You know, the app cannot go down. The platform must stay up all the time to serve those mission critical components. But then how under the hood are we getting better and better at that? How do we change TTPs to be more efficient? Tactics, techniques and procedures. Sorry, I'm gonna jump into acronyms all the time. But how do we improve those different pieces so that over time we're becoming more and more effective at those missions themselves, and more and more efficient at those missions themselves? And so really what we want is we want to figure out how to maintain a certain velocity that we have effective operational agility and this is gonna give us a continuous advantage over all of our adversaries. And that's really where our ultimate goal is, right? It's not revenue, it's not P&L. Those are all awesome, right? We all as taxpayer dollars want those different things. But when it really comes down to focusing on the mission, it's how do we maintain that continuous advantage over our adversaries that are all threatening to attack our way of life? And so you always gotta remember why we're delivering, right? So loss, right? We're looking at better ways to put folks forward. How do decisions faster so that it gets them out of harm's way? How do we more effectively drop ordinance on targets to minimize collateral damage and things like that? These are all the different pieces that we're trying to do as the number one priority but then still go fast. And a lot of times that's at odds. And so that's what we're sort of learning in this whole effort in terms of like Kessel Run, in terms of what we did at NGA, in terms of our other federal clients is how do we learn from this and reduce the friction in order to do both aspects of this at a minimum? We all know this, right? We all have organizational silos. Every single effort that we've done, we come across this. In our own organization, we have this. It's something that's always there and we just learn to get better. And so you heard a little bit about this earlier, so it's kind of nice because I don't have to go through all the stuff because Captain Kruger and Tori did a really good job of articulating smaller deliveries, faster deliveries, less risk, eliminate that stuff. So I don't know if you guys have seen the sawtooth versus the exponential, things like that. I mean, that's really what we're after here is how do we reduce risk through the deployments? DIY platforms, you heard a little bit earlier. Without a unified fielding approach, this is what you get. Everybody goes off and builds their own and then you have to come up with adapters for every single different standard. This is what we're trying to eliminate. Is the system of systems architecture that's so complicated, it requires a large sustainment tail. It requires a lot of troubleshooting and a whole organization stood up to support it. This is what we're trying to get through and be more, be more, I don't want to use agile, but be more nimble in the way we actually approach these different integration points. Because this is what we want to get to. This is really the key, right? The faster you can get feedback from the warfighter, the more the applications resonate, the better the intuitiveness of the actual application is, so that then they can effectively do their missions and then even start thinking outside the box with this extra time that I've got because it became more efficient, I can now think of better ways to do those techniques, tactics and procedures in order to be more efficient in my mission. And it becomes like a cycle that's good, but we'll get to that later. Non-functional requirements. I'm sure you guys are all aware, non-functional requirements tend to be the first thing scrapped when budget cuts come. It's like, hey, it's out there, it's good, it breaks, hey, we'll support it, get it back up and running, but it's not, it's things that, if you're proactive about it, we can get it much more stable, we can get that ahead of the curve in the beginning, we can address more things on the non-functional side, and it doesn't have to be an extra effort and the first thing cut when budgets come, it can be part of the integrated process, which a lot of it, we do some of that with the TDD, sorry, the test driven development sort of focus, and we do a lot more quality types of coding and things like that, but there's even more processes that can be put in place with automated pipelines graduating through environments and going through maturation gateways, like canary testing, like load testing, chaos testing, resiliency exercises to prove that the designs are valid and that they'll hold up under the adverse conditions downrange in the field. The two most important factors, performance and testing. These are often overlooked, and like I said, they're the things that require the most care and feeding in the beginning, but then once they're built into the DNA of the applications going forward, we're able to actually carry that all the way through to better stability out in the edge. And you heard a little bit about this earlier too, functioning acceptable is no longer acceptable. So a lot of this, we go in and we start talking about culture and cultural changes, these things are really important to understand because the way business has been traditionally done, it has been, here's the requirement, did you meet it? Yes, check. There's no qualitative aspect to it. It's we've done the bare minimum and that's no longer good enough. You heard Captain Krueger say earlier about building joyful software. And it's really, I mean, that's definitely part of it, like the joy of the user, but there's also a whole another under the iceberg type of aspect, which is if I can go in and I can pull the plug on a rack and we've designed the applications with the idea that it's a mission-critical application and pulling the plug on that rack will not cause any outages in the mission capability. Those are the type of things that we wanna build up to and the resiliency in the mission-critical space. And this is fundamentally different. Yeah, I get the commercial likes that too, but it's less likely to happen in commercial. You'll have outages and you'll have these kind of things. When you have the threat of actually being under fire and being able to do disaster recovery and shifting sites and conducting mission operations on the next heartbeat, that's the type of things that we need to build into the DNA of the quality of the stuff going forward. So these elements are critical in fielding every delivery and sustainment of every capability going forward. Because like I said, we wanna get to this. The only way we build trust with the warfighter is when they know that when it gets put out in operations, it's not gonna go down. They can't trust that if, you know, I have a sexy new feature that I asked for and it's there, but then it disappears on me midway through and I've gotta revert back to the old way of doing business, that's a problem. And then as that erodes the trust, then it becomes more and more difficult to solicit feedback from them to build better, more resonant apps and do this kind of stuff. So to get back to that cycle where they're providing us feedback, we're building better software, they provide us more feedback and we're getting more and more effective missions, this is the type of thing that has to be built into the DNA of this. Cause we wanna focus on code, right? Platform is always treated as an african thought and I'm a platform guy, so it's fine. I can say those type of things, but that's what you want, right? The ultimate testament of success is when I plant the flag and say, nobody notices me in the room, right? I mean, that's ultimately what we wanna get to. We wanna get it to where you're not even thinking about that, but that doesn't mean that we can cut corners, right? Because safe software is difficult.  I know we love to sound bite a lot of different things from all different levels, but at the end of the day, safe software is tough. It is tough. It requires a lot of thought in how this stuff is deployed. It requires a good understanding of the characteristics of the objectives of the application itself. It requires understanding the environment that the platform is going into, that the app is going into and the types of missions that would be conducted on top of it. And so these things, when you kind of find that empathy with the platform teams and the application teams, you start realizing what is actually important in the delivery and it starts helping you set the priorities on feature development. It starts helping you set the priorities on platform evolution and things that you wanna do in this particular vertical. But ultimately, this is what a DIY platform looks like. Like this is what you wanna get away from because you're not in the business of building a platform in the Defense Department. You're in the business of actually conducting mission operations. And so this is why you come to like sort of a unified platform strategy. It's because it offloads a lot of these different things from you and having to learn all these different skill sets and then keeping track of all the different evolutions of all those different technologies, licensing, supportability, those different types of things. And so once you kind of take this and look at the unified strategy and you kind of take that off the plate, now you can focus on organizing around the mission. And this is really what you want. Yeah, it's crown, it's my cartoon, so it's fine. I don't really have a graphic for it, but these are all the different elements that go into the application. Like if you think about this, in this particular vertical, user experience, warfighter owning the product and providing that feedback, government sponsors and a lot of you guys are those folks in the room. It's how do you organize around the mission so that instead of having these vertical silos, we can get more into a horizontal flow and out to the field faster and faster. And so that's really why we do this organization around the mission. Because remember, we wanna do this, right? This is what we're trying to build up to, but we can't cut corners. There's no work around for IA. There isn't, like as much as IA has always been that long pull in the tent and we're starting to conquer it, we're starting to get through that. You heard earlier about shared application controls that are inherited all the way up from infrastructure, all the way up through the platform, and now you're only dealing with 40 some odd controls. We did not shortcut that. We knew we wanted to go fast, but we did not shortcut the process because ultimately it then opens us up to vulnerability. And that's what we do not want. At the end of the day, we wanna be reliable and we wanna be secure. And then from there, the sectionist comes with all the different features. Continuous patching is part of that piece. Introducing that whole different concept and idea, which was like, oh my God, like you guys can patch like all the time. And there's no end of life. There's no EOL problems like we had in the past where you're still sitting on Windows 98 because we can't get a new operating system out there. We can keep the operating systems up to date across the entire portfolio. From the VM level all the way up through the application, we can continuously patch that. And in working with those MOAs and understanding how those things are gonna roll out and how we still maintain the stability of patching all the time, but being able to canary style those things out and see if they're gonna crash and burn. And if we need to go back to the drawing board and provide another patch, or everything is good and roll it across the entire platform. And in four to six hours, your entire piece is all up to date, which then becomes a moving target for any of our adversaries to attack. I'm sure some of you guys have probably heard our repaving story. And things like that, creating a small moving target is a lot harder to hit. And then you combine that with all the in-depth layer of security and stuff that we put around the platform and networks. And now you have a very resilient system, but also a very secure system in order to move forward with mission critical pieces. And we also are able to then offload resources to the experts. So things like a unified security model. I don't know, you guys ever remember working in the past about how we had to do security before? Every app had to bring their own security. And then every time there was a change or there was a vulnerability, you had to go through and figure out who was using which security model and how that stuff would work. Identity access was the same way. And it became very problematic to do any kind of updates. And it always required long lead times and RE events and all these different pieces just to patch or just to update or just to upgrade some aspect of identity management security. And you had to do it because if you left it, then you have a vulnerability in there. And sometimes those vulnerabilities would sit there for three years, you know? And so what you do here is you say, okay, we have a security team and they will run that piece and we will integrate with that security. And then that shared security model will then move down across all the applications within the platform and in the entire portfolio. And you know what? The developer doesn't even need to know about it. They say, bind to my SSO app and I get passed a resource token. And I know what to do with that resource token because it's talking about roles that I need to develop in my application. And I know what those mean, but I don't know what that PKI stuff is. I don't know what all is, you know what, you don't have to worry about that. That's all gone, right? There's a team that's handling that. We've integrated with that team and now everybody with the roles across the platform can just deal with the actual application pushes. Runtime, same kind of thing. How do we verify what's in runtime? How do we know we don't have an advanced persistent threat that is burrowed into our stuff? We don't care because we paved that stuff. You know, you set your policy guidelines, we can pave through containers. We can pave through VMs. We can pave through entire architectures to make sure that if there is something all the time that they spent trying to penetrate and get in there, suddenly they just got wiped and they got to start all over again. And so pretty soon they'll give up. You know what I mean? If they even get through in the first place. And so the idea is we can even verify at runtime the types of things that are out there and making sure that these things stay stable. Continuous feeling, what is it? Right, that's a culmination of all these different pieces. The resultant outcomes that produce the tangential benefits from the fiscal responsibility that Tori talked about through the mission applications and how we continually provide actual capabilities but without sacrificing anything in the meantime. So now we're ready to talk about this, right? So you heard a little bit about what we call our UCD or user-centric design process.  I think what you find, which is often a understated benefit is that there are no tech manuals. There aren't a need for tech manuals, right? They're very lightweight. Why? Because they told us exactly how you would want to use the application. And so this is the epiphany, right? The idea is that if you tell us how to use it we don't have to provide a tech manual for it. I mean we're going to provide release notes and things about it, but we don't have to provide these big giant tech notes and then it doesn't make sense how they're using it. It's all becoming intuitive, which means what does that mean? It means that we actually look at how do we train techniques, tactics and procedures instead of worrying about in the tech manual how to pull down a drop-down box, how to change my font size, how to do these different pieces that have nothing to do with my job, nothing to do with the mission that I'm executing. It's all stuff that is, it's just been there and I've always had to deal with it before. The other thing that we do is realistic data, right? It's this idea that you can't mock up the kind of stuff that we see. This could be something as simple as real, link 16 stuff or USMTF messages or formats that come over and not realizing that they can go beyond a certain character that we thought was there, right? That then suddenly mangles displays on the screen. It's from that level of detail all the way up to the size and scale, like tanker planner and how many tankers are in theater, how many planes are they trying to refuel? How many targets are we talking about here? Global operations. What does that actually look like and how many things do we have to keep track of and do this type of thing? Using realistic data gave us some new insight into that mocks don't always do what we need them to do and so there needs to be a realistic testing environment in order to make sure that when it goes out into operations it's gonna perform as expected. And again, the ultimate goal, training time shifts from tools to mission function. I mean, there's no better thing than that, right? Like the idea that now I'm doing my job and that's the only thing I'm doing. I'm not doing all these other things that I have to do in order just to get to being able to execute my particular job. And that's the intuitiveness of the user-centric design and how that payoff works at the end of this. And then, you know, never settling for less than direct feedback from our users. And this is a big thing, right? This is, you know, in the past and this is not a knock on anybody, it's just how the organization's worked. You had a requirements office and you heard what Tori has said before. You have a requirements office that gets a requirement and it vets it for three years. Then it goes through an acquisition process that tax on another piece. And the thing is is what that requirement was is not what it is today in the mission space. And so by not eliciting feedback directly from the actual warfighter, then you don't know if the actual mission requirement is still valid. And so the same way you can save money like creating a more efficient tanker planner, you also save a whole bunch of money on requirements that you can sunset that you never even started building because there's a implied debt that's sitting out there. You know, all my unsatisfied requirements all add up to some dollar amount. And if I'm just getting chunks of money from Congress to just execute those requirements and 80% of them are no longer valid because they're stale, I just saved whatever 80% of that budget was, right? So there's also this addition by subtraction that you get by having this direct feedback from the user. So there's the cost of delay and then there's the cost of implementing the wrong thing. And that's the other thing that we get or we minimize when we have this direct user feedback. And then quality, right? We don't wanna sacrifice. Developers and testers are not a different function. This is the new way of doing software, right? There's no test organization doing manual checklist anymore. And then that goes into a black hole and then you provide feedback and you don't understand the context. You don't understand how they actually ran the test. The feedback is maybe not useful or by the time you got that feedback you're already three releases forward and you already discovered some of these problems so you wasted manpower. So the idea here is that when you switch to more of a test-driven development model your developers are the testers and the testers are developers which means that if you think about this from a DOD perspective with test organizations you have a whole bunch of billets sitting out there. I know this is controversial but you'll have a whole bunch of coders that are sitting out there when you start rethinking about the test-driven organizations and making them into TDD coders. And so that's a huge set of billets that are outstanding that you could actually start converting and make them part of the fight and being able to deliver these capabilities forward. It's all about the journey, right? So we do journey testing in mimicking user base, how they do operations, these type of things. Quality and coding is an art form. I'm sure you guys give you a chance to read the cartoons but this is what you see all the time in the past, right? This is the idea, is you have a quality assurance organization that's separate from your actual coders, that's separate from your actual testers. And so what ends up happening is you get these type of things where it's like nobody understands what the actual code did. The QA folks don't understand what it's there for, they can't really articulate why it shouldn't be there. You've got coders that built something that they think they understand what the requirement is but then it doesn't function in a production sort of setting. And so you have all different kinds of disconnects between these organizations which is another reason why we organize around the mission and do this. And then this was a big challenge specifically with the Air Force, we didn't have as much in the NGA part but overcoming that transient workforce, right? Transient being loosely user. Right, and so the idea here is that, it was something that was just, it actually just happened to be a side benefit, right? Because we do pairing, because we do pairing it really illustrated the strength of pairing that you don't normally get to see in a commercial setting is the idea is like, if you have people shipping out every six months, if you worked anywhere else that did not do pairing, all that context would be lost and you'd be constantly on resetting. But not only did the Air Force survive, they ended up thriving on that pairing model such that there was, people would come and go, guest appearances, people would come and go in six months, people would come and go in a year and they were still able to maintain that consistent context across the entire effort. And that's a testament to the pairing model working and then the Air Force fully understanding the power of that pairing model and how to apply it to the adversity that they were experiencing with that particular workforce. And then of course, the most difficult part, right? Velocity to the edge, a lot of great ideas end up on the shelf.  It's really getting over that last hurdle to get it out to the field and that's where we focused on first was what would it take to get it? How do we start at the field? How do we build the pipelines back so that we could turn on like, the moment we can get one drop means the moment I can turn the faucet on full blast and get capabilities delivered out to the field. And so that's really what we were looking at is how do we overcome the most difficult part, the velocity to the edge piece? And then why? So here's a whole another benefit to why we do this and why we offloaded the platform, the invisible hand of the market. So how many Silicon Valley efforts do we have around the country now? Boston's a hub, Silicon Valley is a hub. You know, here's tech solutions popping up all over but the reason why those things aren't getting into the government is because what you have is you have this barrier to entry.  So two kids in a garage shop that are brilliant at building pathing algorithms will never get that in front of the warfighter in the past. But now with this and offloading all of these different elements of how to graduate code into production and into operations, how to offload the security identity pieces, they just have to focus on what they do well. And what this does is it lowers the total cost of ownership of capability development. It means that if this isn't performing well, some big defense contractor over here, five layers of management is costing me this much money and I can do a fly-off with a two-person shop and that actual algorithm is just as effective but yet the cost is super low because they don't have that and this is going to scare big defense contractors. Sorry, Johnny. Anyway, so like the idea here is that when you look at that, that is what we want as taxpayers. We want the best ability, best capabilities going out to the field to support a warfighter. But all those sides that I showed before about not sacrificing anything, that's what we have to maintain. So this is a perfect model for that because by offloading it to the platform, you've done that, right? You've already ensured the maturity is going to be there by the time it graduates into operations but now you're able to source five different pathing algorithms. Have them fly off. Which one won? Which one was the most value based on its cost? And then the support and sustainment pieces come through because why? If I want them to continue to use my pathing algorithm, I better keep updating it, right? Because the next one's right behind me and now I've got a competition for it. So it's awesome, right? It creates this whole invisible hand in the defense sector, which has been kind of non-existent since the 60s, right? Because the big defense contractors kind of boxed everybody out and this is the gateway and everything became so complicated to get in and now we've eliminated that. And so this is an exciting new time pioneering where people can set up two four-person shops and build businesses and have a nice little business being service marketplace inside a cloud foundry and these type of things and they can continue to sustain and maintain this. And what this means is simplifying delivery means the overall TCO goes down, total cost of ownership. And that's gonna be pay huge dividends in the long run for us as U.S. citizens and taxpayers in the Department of Defense. And so ultimately what we learned from this is that this model works, right? Have you guys heard the phrase watching the vapor trail? So this is where a spotter is looking because snipers are shooting so far you can't actually see the rounds on target. So this is making those micro adjustments here and there until we can actually hit on target every single shot. And that's really what we're trying to do here is continually to learn and continually to evolve but make sure that we're constantly giving that micro adjustment so that we're always trying to get those shots on target. And that's it, any Q and A?


Did he shoot? Yes, I could shoot. How good did that feel? No, it was pretty cool, you know. I would just tell everybody, that's just how I talk, though. You know, I know he probably didn't mean to disrespect me, but you know, the heat of the game, and that was just my first best phone when I heard it. So I was just like, that's it.


All right, so it's 2.30, so we'll go ahead and get started. I've got the dubious honor of following lunch on the last day of the conference, so, you know, hopefully no one falls asleep, bear with me. So thank you all for coming. We're gonna talk about extreme ops, kind of a play on extreme programming, but how we do platform operations at the Home Depot. I should have probably worn like an American Gladiators costume or something like that. Probably would have worked out better, but. Actually, I forgot to put my married name on this. I'm Kevin Nigarashi Ball. I'm a staff software engineer for the Home Depot. I work with the App Platforms team. We're primarily focused on delivering platforms to our software engineers throughout the enterprise, one of those being Cloud Foundry. You'll also notice in the title that it does say software engineer. That's kind of important as we progress through this talk. We see ourselves as a software team that happens to deliver infrastructure versus an infrastructure team that actually just happens to write software. Other titles for this slide could have been presentation, could have been, you can do it, we can help. More developer enabled, more saving, more doing. Or, let's just let's do this. By the way, I'm Dan from Bringing More Home Rebuckets Home by my husband. So this is the closest I could get with one. So give you a quick glance of where we are as a platform team right now. We are running six foundations and growing. Currently we have about 1,200 engineers. That depends on where you wanna look for that number. It could be higher or lower. So I tend to go off of conversations. And then in our largest foundation, we're running about 1,500 applications with about 4,000 plus instances. If you look at non-production, double or triple that number. Don't really count those. So let's go back. So our journey with Cloud Foundry started about two years ago. And we've, as a team, we've had a couple of different application platforms. And adopting Cloud Foundry, we, and not to go off of the buzzword of the day, but started our digital transformation as part of, kinda coincided when we were bringing Cloud Foundry online. And as part of that, we actually started to look at what's really impacting our software engineers. What is keeping them from delivering value? And there was a common couple of themes that came out. A lot of it came around matrix people. So actually having to reach out to someone else to do, reach out to someone else to do something for you. And not being empowered to do their own thing. And really around this came a couple of different concepts of, or different pain points. One of them was tickets. And one thing, and you may have seen the tagline, no cues, no tickets. And really, we took that to heart.  You may be asking yourself, how is this possible? How in the world can you go and not have a ticketing cue system to do work? Isn't that how I monitor things, how I know what my volume is? Am I getting better or worse? But it also kind of detached us from our guys. So anyone remember Tom from Office Space? Takes us, no that's what tickets really do. T, you know, Tom works really well. I take the specs from the customers and I give them to the engineers. Engineers, do we like to talk to people? Sometimes, sometimes I love just coming in, putting my headphones in, going for the day and being fine with it. But Tom's big caveat for this was he has people skills. And really, you know, this ticket's removed that ability from us having people skills. No, because how many of you had to fill out a ticket and then actually have to have someone call you to figure out what your details were? Yeah, anyone ever had a ticket like that was self-service the entire way through never had to talk to the person on the end? Yeah, never works. Always have to do that. So trust me, came from a help desk background, doesn't work. You have to talk to someone. So why should I have you put fill information into a ticket when I can just pick up the phone and call you or you can call me. We'll get to where that doesn't, that kind of breaks down how we interact with that. So we took on a thing called conversational support. Really, you pick up or talk to me and we'll figure out how to help you. This is not just picking up the phone, this is a lot, we're heavy users of Slack. Some of this is an email. A lot of this was also just road shows and going and actually talking to people and going to the different floors of the development teams and being like, hey, let me go to your stand up and see what's your pain point right now. Or, hey, this has been really cool in Cloud Foundry that we're introducing, here's how it might help you. We also found this kind of worked into some workshops. So when you're new to Cloud Foundry, how do you see a push? How do you look for your logs? Things like that. And we found out of those workshops that teams really had issues learning or not only did they not know Cloud Foundry, they didn't know Git, they didn't know Slack. And so if we hadn't been talking with these people, we wouldn't have known that those are issues that we need to help cover. And so that workshop actually morphed into a kind of a modern development workshop of how do we talk about, hey, here's how you engage with people on Slack. Be kind, don't channel everyone in a massive channel, please. And so right now, to look at where we are, like our main Cloud Foundry channel is where all our interactions are, where our community lives, has about 1200 developers. That's kind of where the number came from. That comes up and goes down. But if anyone's familiar with running Cloud Foundry, it's not just, hey, it's the foundation, that's what I'm running. There really is a whole team, especially in an enterprise that you're gonna interact with.  And they need a place to talk. And really in the big main channel is a little bit too much to talk to these people. So we actually spun off another channel called PCF Operators. This is where the nitty-gritty, hey, let's talk about SNPs and NATs and all the other fun stuff happens. And we leave that, but it's also, you'll notice it's not a private channel. It's a completely open channel for anyone to join. We have developers sitting there, or engineers sitting there, watching us talk about the deep, diving entrances of Cloud Foundry. One thing that comes out of this though, with Cloud Foundry's conversational support, and this really comes from extreme programming, trying to do face-to-face conversation, trying to get that feedback loop shortened as fast as we can. We'll get to some other ways we do that so we know things are working. But with that, anyone ever been overwhelmed by messages and Slack? That really happens a lot. I became quickly the PCF dude, because I'm the one that's been working with the Foundation this long. Getting 20 DMs, 30 DMs in a day was easy to happen, and it's like, please go talk in the channel while I answer you there so someone can search. But really, the whole team would spend, you could spend time just talking, just doing support items. And so we introduced the concept of an interrupt support pair. So we already pair program, so why don't we just pair support? It's not, you know, it's maybe not the most glamorous thing in the world, but it does work, especially as you're growing a team. As you're growing a team of operators, you want having that anchor, bringing a junior person from the team and actually work through this support. Hey, here's how we go through Bosch and actually troubleshoot these things. Or here's where you find logs and where they're stored on the Diego cells. Here's where they're stored and how you look for those.  Plus, so in the top of our channel, we put, hey, here's who's support this week. Go talk to them, bug them, leave the rest of the team alone to be iterating on deploying new features in Cloud Foundry or to actually, as I said, we're a software team. We build things that go around Cloud Foundry. One of our big things is we build things that other people don't want to build. Getting identity information and things like that. It's kind of nice to have a centralized team that can work on those things. And the support pair isn't a horrible thing because you're kind of not designed to have anything coming through. You're not on the roadmap, you're not in Tracker, you're not the one trying to actually get work done. So you can either work on some stuff on the side that's not as important or some of those kind of out there projects of what if we had time to do this while you're sitting there watching Slack? It gives us a little bit of flexibility that we normally wouldn't have. But again, 90%, so you have all this conversation, you have someone actually handling that. But what you really see are a lot of the questions about Cloud Foundry or they just happen to be that I'm running on Cloud Foundry, so I'm going to ask the Cloud Foundry guys because they'll know about it. As a lot of, hey, how do I do this in spring on Cloud Foundry, which Cloud Foundry is just that little bolted piece on the end. And really that's where we started to take it from being this little support pair to building our community. We do a heavy, we have a huge community, it's all 12,000 developers. They range in there from people who understand Ruby who are hardcore spring guys. There's some awesome monitoring guys in there that are like, oh, this is how you can find these things if you went to this report. A lot of those questions that I wish Slack gave it to me that I could show the graph of how much our team doesn't talk in there anymore and how many of the community are contributing. It doesn't, so. But really that's where the value is because I can answer your question or another question. But if you post there, we have a conversation, here's what you're running into. And then two days later when someone else joins the channel and has to figure out what that question is, it's in there in the search feature. Trust me, that's usually my first question of have you searched Slack, have you done in Cloud Foundry and searched? Because usually that question's already been asked. And like I said, there's a lot of times where we have cases where people maybe have you done this in Cloud Foundry? And people can help with it. And really that's where we started shaping the community into aspects of what's the difference between help, support, guidance, advice? Our job as operators is to operate the platform. That's where our SLA comes from. Our goal is to ensure that the platform's up and running, serving traffic and your apps can push. But like I said, we're software engineers as well. And so as part of that, we may not be able to support you because I'm not the one sitting there with your application, but I can help you as a fellow, someone getting to go, I can help you a little bit with that. I can tell you where here's some articles that I found that really are beneficial. We have other people who post guidance of things like maybe you shouldn't try that because horizontal scaling's better than vertical scaling. Let me tell you why. Why that doesn't work really well when you try to place massive VMs. But that's one thing we really, we get into what we call community management of now we've built this massive community. How do I get people who aren't my team to answer questions for my team? And also how do we get the questions that really do need to matter? Like, hey, I'm seeing this huge error. What does this actually mean? But you can't talk entirely. You can't sit here and actually talk to people in Slack and be like, oh, that's great, it works. I can solve world problems. Really caught in everyone should have knows Cloud Foundry is a big hairy beast sometimes to go find where an issue is. So how do you go find that? How do you do that when you're getting pinged all the time as a support pair of like, what's the problem? So we lean heavily into automation for everyone. This is getting back into the whole extreme programming of like, again, fast failures. One of our big things is how do you know the platform is working? How do you, when someone says I can't push an app, how do you prove that? How do you prove that it's you, not me, you know, that kind of thing? Usually works the other way around, but. And so one thing that we do is we constantly run how we prove that is we do it ourselves. So we run tests every five, 10 minutes on the platform that exercise everything.  If you can't push code because this build pack is offline, we know about that. And I would rather get woken up by that page with a little, hey, this is what's working. And then someone's saying, yeah, I can't push an app. I don't know why. Please join this call. That's like, no. The other thing we wanna get into start doing is actually automating upgrades. I like to say I have artisanally handcrafted foundations right now. They're, you know, they're unique. They're snowflakes, they're pretty. Love them. They're not that unique in that snowflake, but it takes like a little bit of time to actually do upgrades. Wouldn't I love just to have a pipe wine that said, hey, there's a new version of this out. Let's go send that to Sandbox and let's take those smoke tests that we had and let's just run that on there for like five hours. See what happens. And then if everything looks good, then yeah, maybe we're gonna send it to non-broad. That's my pipe dream over the next couple of months, but we gotta get there. Because as you said, we're growing, as I said, we're growing. When you start growing foundations and having more and more, it gets really hard for that handcrafted artisanal product to happen. And you have to start stamping out constant, you know, iterations of those. The other thing we lean in heavily is cell service. This comes in many forms. One of the biggest ones is from, well, I wouldn't say from day one, from day one, it was like, let's apply traditional platform ups, let's lock everything down. You can't get in and you have to ask these people or you get a granted access. We did that for a little bit and it works. People aren't in love with it. So if you look at our non-production environment, we said, hey, what if you just, all you had to do is be able to authenticate and you could create an org on your own. Don't want to talk to anyone. We'll give you space, we'll give you two gig. Here, go play. And we saw adoption skyrocket as soon as we did that. There's good things and bad things about resources being used, but obviously it got more people on the platform. We get into, when you start getting cell service, we started seeing, hey, how do you do some of this tricky networking stuff that you used to have to talk to someone to do? Hey, let's just, again, for a team that writes tools for people who don't usually want to write tools, let's go write something that deals with that networking layer and automates that for you and say, hey, instead of you going to fill out this Excel spreadsheet, which we know we all love Excel for some reason, I don't, I've done horrible things in Excel, trust me. Really, really bad things generating like 35 page web pages for a morning report. Yeah, not pretty. But let's automate these things. Give, when you give the developers that access and just say, hey, go and you get out of the way, Cloud Founder generally keeps things running and so now you're not asking me for help and so I can start iterating on those hard to solve problems faster. Or I can be able to get those, those upgrades that take, right now, it'll take a while because they're increasingly handcrafted a lot faster out into the world. So, a lot of this comes down to the point of how do we know it's working? How do I know that this is actually something that's anecdotally working? Because I've now removed my whole ticketing system. That's traditionally gonna help us background. I can take tickets, I can plot them, easy to see. I know something's actually working. A lot of it does come down to anecdotal evidence. It's going and talking to people. But if you talk to them, again, going back to conversational support and we're kind of feedbacking our own stuff here, is does this work, does it work for you? The overwhelming experience has been very positive. It's been a very plus, you know, we like it because we don't have to deal with that ticketing system. But most developers like it because they don't have to deal with one either. They don't have to deal with going and asking, you know, they ask for quota increases instead of, hey, let me give you initial space. They ask for, hey, wouldn't it be great if this club vendor would do this? And it's like, it's coming in this release, awesome. Or you start getting out of the small problems of let me do this for you, to how do I do these new, cool things? And we've really seen a huge adoption of those things where people are talking more positively about a platform instead of, hey, this old legacy thing that required this financial code and this thing to get working, works really well. So that's all I have in the spirit of being conversational support. I would love to take questions, talk about what I can talk about and how we've done things and where you might have questions from. So I'm happy to open that up to the floor. I have a microphone. Yeah, so the question was, how do we run a health check on the foundation? So we have a couple different things. We run a modified version of the CF acceptance suite that actually runs smoke tests. But what we've done is we've actually broken that out into a pipeline that we can run concurrently. And so we actually test, hey, are the routers working? Is the authentication, is UAA responding? Can I off? And then we actually do a full end to end app deployment, scaling, our logs coming out. And so from that, we can get alerts on those and see if it's actually working. As well as some of the administrative stuff that we do of like, hey, can I create orgs and spaces? Can I remove them? Can I change quotas? Things like that. We also have a set of status bots that look and look for latency on the network. So if we have a component that's getting overloaded, we'll get a latency trigger that says, hey, this clock controller is overloaded right now because you decided to move 400 containers at once. Or you're draining 20, 30 cells. Crap. That's what we use. We use a lot of different combination and we're looking to expand. Like anything that would require a human to look at, we want to automate as much as we can and have that run as part of the smoke testing suite as often as possible. Yeah. How long does your team last? So if you look at the core team that supports it, it's about six or seven people that support the platform. That being said, there's a larger team of about probably 20. If you add in networking, infrastructure, things like that, but those aren't part of our team. They're just very close partners. I would say start by just opening up a public channel. Like if you're heavy users, if you're easy to go slack or even sending them an email address works, but that's kind of still that asynchronousness. But a lot of it's just talking in the open. And that can be anything publicly, it's just being able to talk to people. One thing that was always, it kind of frustrating is when you have a ticket, you don't use submit the ticket. And then I have no idea the two or three hours that happened on the back end where there's probably four or five different discussions going on until I get the response back to you. If all that's happening in the open where it's like everyone can see that, that if that does take four hours to solve something, at least they're seeing this is the actual progress that's being made. And you're introducing people to actually like, I'm just the front for the core cloud foundry. There may be, this may actually be something like in the networking layer. So here's the networking guy. So if you have these kinds of things in the future, you may not need to talk to me, just go talk to the networking people and just get everyone to talk more in the open. So, one second. Yes? How do you keep- How do you, for what? Do real work. Doing real work. So we're kind of fortunate that we're kind of separated off a little bit, but when we weren't, we actually, you'd have like the support pair sets here and there's a little, you know, like here they are, or if they would walk up, it's like, go talk to them. I'm in the middle of doing something. It's still a little- The walk-up side. It's still, so the inner pair would do both. So they would do walk-up or, mainly like I said, we're heavy users of Slack, so Slack support. But, so if they're chatting with someone in Slack, if someone walks up, it's pretty much, you're gonna have to hold a minute or I'm talking to this other person. Usually it's never where their heads down completely where they can't kind of give them at least five minutes. But there are some cases where it's like, I'm really, we're into a weird issue. So can you just give me, or can you come back in like five, 10 minutes or where do you sit? Let me go find you and go and talk to them. Because usually they're gonna rope in a couple other people as well. I always love the walk-ups that have like four or five people show up at the same time. It's like, I'm outnumbered. Help. So, yes.  Yeah, I can just sort of agree. So as far as what's open and available to them, so non-prod is open, which is our development environment to everyone to do work. The production environments are locked down. So you can't just log in the production and push code. There is change controls around that where you do have to have documentation in order to go to production. We prefer that going through some kind of CICD pipeline. Don't get automated, don't manually ship it. We don't necessarily restrict that. But we try to allow it as much as we can, but yeah, there's for compliance and regular reasons we have to have. No, you can't touch prod. But that's not having, on our legacy system, going to non-prod required another team. So being able to talk directly there and push my own code for my local workstation there, as a lot of people have really been beneficial by that, so. Go for it. Yes. So then people can see the data? Yes, so we do maintain, so we have this one Slack channel where usually all this happens. We have spawned some out and gone back in. A lot of, if it becomes a question that we get often, we actually have a knowledge base that we post those things to. We also use pinning a lot in Slack, but if ideally you don't want your pins to grow, so if it's something that's constantly referenced, we'll pull it out of a pin and drop it into the knowledge base and remove the pin. So we do have two places you can search for that information. So if we're not able to solve it, it's kind of fun, but no, so if it goes to the support parent, they're not able to solve or they need to pull in someone who's more staff within, they can pull us off to help them. If that's not, we will reach out for help from the community or from different people to get those, it just varies on the case by case. A lot of it is low level, like hey, I need this kind of thing or how do you do this? So. Non-prolucidation, non-functional testing environments and all that stuff, so how do you organize all that? Yeah, so we do full separation of a production and a non-production foundation. As far as orgs and spaces, we don't necessarily dictate how teams do that. We prefer to have self-organizing teams, so if they want to organize by their business unit, they can or if they happen to be like, this is the application name, they want to call that org that. That's completely up to them and they can define spaces inside of there as they wish. We try to not to have an opinionated stance on that. We run PCF everywhere, so we run Pivotal Cloud Foundry. Do you find it difficult managing all the different? Not necessarily, like I said, it is right now we have to automate a lot, we don't have a lot of automation for upgrading, but as far as from our administrative side, we've got it tied back into some of our identity providers that we don't have to worry about, hey, this is how you authenticate in some of those things. And it's not really, it's just figuring out where you're going and where they, that's really our biggest question of opening it up is, hey, where are you asking this question from? Like where are you, what foundation? So we can target the right place to look. Same thing for a ticketing perspective, like hey, can you tell me where this is? It's okay, I probably can't say. Where are we running Cloud Foundry? What are we running it on? So we run, we run on-premise and private cloud. We run on in-house and in our own data centers on private cloud. So you're good with that, sorry, Tony's my manager, so I'm getting the green light of like PR, can I talk about that? So we run on vSphere on-premise. We are looking at public cloud as well. But for us Cloud Foundry, our goal to delivering to developers is Cloud Foundry to Cloud Foundry. You just need to tell it which foundation you're talking to. We kind of take care of everything under the covers. Yes. It's something that we're definitely considering. It's depending from our auditing and regulatory controls of is it good enough or are they comfortable enough with it? But it's definitely something that we're considering. Yes. That's kind of an easy way of saying our capacity is shrinking or our needs are growing and we need to add that counter to the team that's overseeing our understaff or something like that. When you've lost that signal and now you're in fully conversational support, number of slack messages doesn't indicate complexity. There's no time to resolution unless you're doing something on paper or something else. And you've found ways in order to try and figure out how to balance that meeting to staff up to the right level and what's the signal that you use up that just aren't really going to work? Yeah. So the question is like how do you, since you've lost that metric from ticketing, how do you actually go and actually figure out how do you monitor as the team healthy and things like that? Really for us, it becomes, you start monitoring actually the team itself. Is the team happy? It becomes kind of that. And that's kind of a weird kind of DevOps emotional kind of thing. But it really is, you can tell when the workload's getting too much that the team kind of changes slightly. And so you kind of, and you can tell like, okay, this is I'm done with being support or if the team's getting bogged down on other development tasks and the support pair isn't rotating. If they do, you know, you just still get these people you're gonna do support right now. You can kind of start looking like, so we're understaffed. Like things are underperforming we need to be able to start rotating again because we're holding anchors on other projects. But really you can, it's one of those emotional things you can kind of tell as the team gets overloaded. Yes? Those are lots of devs and lots of apps here. We are, so at the moment, we have not. So that is definitely something we are looking at more of. I definitely from a showback perspective of here's how much you're costing and hoping that helps people realize like if we could write things differently maybe it'll be a little cheaper. But we really wanted to drive adoption as our biggest thing of getting people to migrate to Cloud Foundry. Because we saw that as a really strategic advantage. Yes? Fourth pair, so I'm assuming that means two? Usually two. How many developers and how many app teams are you supporting with that? So that's about 1,000 to 2,000 devs. And it's not always a support pair. You can sometimes just, they fly solo depending on team strength. But there's always usually someone dedicated for support. We do run some data services concurrently with it. But primarily we're focused on runtime. So our team is kind of a sub component of a development or enablement group who kind of manage the other side of the house with DevTools. So they can kind of have both groups up and go to that. We really as far as excellence go to trying to have other people like talk to the other people who are doing this. And there's usually from our channels, we know who's the spring expert or really vocal in spring. Or hey, these are the IaaS people who are really good at running and or here's where your Oracle people are. We have a lot of Slack channels for people to do that. You're about to try to put some stuff that I designed that as an application. Do we have anything like that? So for our team, we support Cloud Foundry. So we don't actually operate the application layer. That's actually up to the application teams to either do their own operations or have their own ops team. Our goal is to say, hey, Cloud Foundry is running, your apps are being served. So we don't necessarily specify like, hey, this is the foundation you should be on. Or not sorry, it's not foundation framework that you should be using or a language or whatever. It's what makes sense for your team. Yeah, we just run it. We make sure it's up and running and then bring your apps to us. We'll host them. So hit it off, but it's past their clock. So thank you so much.