Great. Thank you so much, Mike. And hello to everyone. Thank you for joining us for today's program on how to protect against risks to tribal cultural heritage. This program is made possible with the generous support of the Department of the Interior. My name is Jessica Unger, and I'm the Emergency Programs Coordinator at the Foundation of the American Institute for Conservation of Historic and Artistic Works. FAIC undertakes and underwrites programs and initiatives to advance the conservation profession nationally and internationally in all its facets and applies the expertise of the profession in addressing global artistic, cultural, and historic preservation priorities. FAIC promotes the preservation of cultural heritage as a means towards a deeper understanding of our shared humanity, the need to express ourselves through creative achievement in the arts, humanities, and sciences. We honor the history and integrity of these achievements through the preservation of cultural materials for future generations. Today's program is the second in a series of three webinars that will discuss the risks that can potentially affect cultural resources of tribes. Cultural resources are defined as the aspects of a cultural system that are valued by or are significantly representative of a culture or that contain significant information about a culture. A cultural resource may be a tangible entity or a cultural practice. Tangible cultural resources are characterized as structures, archaeological resources, cultural landscapes, museum collections, archival documents and photographs, sacred sites, and ethnographic resources. As you might expect, this series will primarily address tangible cultural resources and how to protect them. Yesterday we discussed the principles of risk assessment and how to size up the risk to collections. You can view a recording of that program by following the link that I'm about to share here in the comment box. Today our program will focus on collaborating with emergency professionals. For those in charge of preserving and protecting cultural heritage, the protocols and language of emergency professionals might seem incredibly foreign. However, it is essential to begin building bridges with the emergency professionals who will come to your aid during an event. Collaboration will help ensure that you have full access to the resources available and that your collection priorities are clearly conveyed. On Thursday, we will learn about how to relay all of the information regarding risk assessment and collaboration into a preparedness plan. All of these programs are designed with unique needs of tribes in mind. For today's program on collaboration, we're fortunate to be joined by Patrick Vacha of the Bureau of Indian Affairs. Patrick began his federal service in April of 2003 with FEMA at the Mount Weather Emergency Operations Center in Northern Virginia. As a member of the Mount Weather Police Department, his roles included physical security, investigations, emergency management and intelligence, including time on the National Joint Terrorism Task Force. Patrick was initially hired as a physical security inspector for the Indian Affairs Office of Homeland Security and Emergency Services, OHSES, in June of 2007 to implement the National Infrastructure Protection Program, NIPP, for Indian Affairs. In 2008, he was selected as the Indian Affairs Emergency Management Coordinator, and during a realignment of the emergency management and continuity functions within the Bureau of Indian Affairs, he became the Deputy Chief for the BIA Division of Emergency Management where he currently serves. Patrick, we're thrilled to have you with us here today. I'm going to go ahead and move the slide deck over so that your presentation is ready. So whenever you feel comfortable, go ahead. Great. Thank you, Jessica, very much. Really appreciate the opportunity to be here today. Thanks to you and the folks at FAIC and continue this partnership we've had since, I guess it's been 2012 when we did our first risk evaluation planning program at one of our Tribal Assistance Coordination Group National Workshops. So good morning, good afternoon, everybody. We're going to start as we roll through the slide deck here with a brief overview of the BIA Emergency Management Program as a whole, then do a brief overview of the Tribal Assistance Coordination Group, which is really how BIA and federal partners engage tribes during emergency incidents and events, particularly when they're not rising to the level of the Stafford Act Declaration or significant FEMA involvement. And then we'll get into a deeper, some of the specifics of how emergency managers and Tribal Cultural Historic Preservation folks can work together for planning to protect cultural and historic artifacts for tribes, which as we know are some of the most valuable things that they possess. So we'll go ahead and get started. Again, I'm Patrick Vachis, Head Caesar. Unfortunately, couldn't join us today. He is the Chief of the Division of Emergency Management in Washington, D.C., and I am the Deputy located down here in Nashville, Tennessee. Here's our mission. We know that with 567 federally recognized tribes, emergency management isn't always a very strong priority, unfortunately, because of funding and numerous other things. So we try to be as efficient as we can to help tribes with planning, response, recovery, technical assistance, anything that we can bring to the table to help tribes be better prepared for emergencies and disasters. Our responsibilities, policy at the national level and which also filters down to our regions for preparedness planning response. We also do continuity of operations for the Bureau of Indian Affairs headquarters. We represent the coordinating interagency emergency management activities related to tribal affairs across the board. And we coordinate not only with the Bureau of Indian Affairs and the Office of the Assistant Secretary of Indian Affairs, but we also support the Bureau of Indian Education, which has, I believe, the count as 164 schools nationwide. We are co-located within the Office of Justice Services within the Bureau of Indian Affairs. As Jessica mentioned in my bio, it has not always been that way previously. The emergency management function was in the Office of Assistant Secretary for Indian Affairs, which is a policy piece. So by moving over into Bureau of Indian Affairs proper, it allowed us to be able to not just spend funds on personnel, travel, those types of things, but also operational dollars to be able to provide response activities. And we are a headquarters team that is pre-deployed to the field. And here's where everybody's at. As I mentioned, cities in Washington, D.C. I'm in Nashville. Josh Allen is in our western region in Phoenix, Arizona. Susan King is in our northwest region in Portland. And Jerry Keener is in Muskogee, Oklahoma. Some of the things we've been working on really quickly, the Presidential Policy Directive 8 Refresh as part of DHS FEMA and kind of the whole of emergency management across the board, as are most other federal departments and agencies engaged in, specifically the Tribal Coordination Support and so I'll talk about that briefly. We had a four-corner summit out west, United South of the Eastern Tribes, Impact Week, some recent, very large-scale exercises, our continuity exercises in Washington, D.C. and then Alaska Shield a couple of years ago. Deployments, the first ever during Northwest Wildfires, Bureau of Indian Affairs Emergency Management was supporting the Bureau of Indian Affairs Office of Wildland Fire, which was a very unique situation providing all hazards emergency management support to the wildland fire suppression activities that were going on. Josh Allen was at the Navajo Nation for almost six weeks during the Gold King Mine incident and we constantly provide technical assistance across the board, whether it's to tribes or other federal, state, NGO partners. So this is Sid's slide deck and this is the only picture in here that I actually do remember what this was and we thought this was rather clever that if someone calls and you're on the boat, all you have to do is say, hey, I can't talk right now, I'm in a meeting, so I do remember that one. So I mentioned the Tribal Assistance Coordination Group, an ultra-quick history post-Katrina Indian Health Service where Sid was the chief at the moment, or excuse me, at the time, who is now with us and then the Bureau of Indian Affairs individual that preceded me realized that there was a lot of things going on down the gulf, a lot of folks helping tribes but BIA and IHS Indian Health Service weren't really talking. So they started that dialogue at the headquarters level to try to help make the coordination better at the field and regional level. At some point they realized, hey, FEMA's got a role to play here, so maybe we'll ask FEMA to become involved and they had had a new tribal liaison with an enhanced role in FEMA headquarters and that was really the nexus of the TACG. So starting with three folks, having a couple of meetings has turned into our monthly calls, which have anywhere from 50 to 100 people participating and we just had our 10th annual Tribal Assistance Coordination Group workshop back in November. So we're very pleased with participation and how we've grown over the years. So I mentioned partners from all levels of government, tribal, federal, state, county, local, non-governmental organizations, volunteer organizations, active and disaster, so across the board what folks can bring to the table to help a tribe during a specific incident. We are identified as a new coordinating structure in the National Response Framework and as part of the Tribal Coordination Support Annex and Emergency Support Function 15. So our goal is to provide a focus point of coordination or one stop shop for tribes being impacted by emergencies and disasters. Tribal Emergency Managers, a lot of times, are multi-hatted individuals.  So having to make four or five phone calls for them when something's happening is difficult as well as having to receive four or five phone calls the other direction, asking what help they need or what help can be provided. And so we try to connect folks as a single unit to deliver resources and capabilities to meet unmet needs as quickly and efficiently as we can. As outlined in the NRF, the TACG is led by the Bureau of Indian Affairs Emergency Management Program. And so we engage initially, obviously, within resources within BIA, the Department of the Interior, and then to the rest of the interagency and the rest of the appropriate partners. That's the PNU Mobile. So our national workshop I mentioned was back in November in Tulsa, Oklahoma. One of the highlights of the program was we did a risk evaluation and planning program for tribal cultural institutions. This is the second one we've done at a TACG workshop and we did a very similar program last year in October, I believe it was, at the University of Nebraska, Omaha. So stressing the importance of planning for risks, trying to mitigate risks to cultural and historic artifacts, and then working with as many people as necessary, including emergency management and first responders, to make sure that those plans not only make sense, but can be carried out in the resources that are necessary and available already when they need to. How do tribes request assistance? We always start with an initial phone call or an e-mail followed by a written request from tribal leadership. That initial request, and you've seen a little more detail here when we talk about the scoping call, we want to make sure we've unfortunately had situations in the past and it's been well in the past now, which is a good thing as things have smoothed out, but we would start to engage with someone, a representative from the tribe, who was working on the incident and working hard and working diligently, and for whatever reason, as we started to talk to tribal leadership, that wasn't the individual that they felt was best suited to represent the tribe in this capacity. So the reason we ask for some sort of acknowledgement from either the tribal leader or tribal council is to make sure that we're not seen as meddling in tribal business when it's not needed or wanted. So once we get that initial request, we do have a brief scoping call with appropriate partners to kind of assess the situation, see how things are going, and then a lot of times that may be expanded into a larger call to talk with more partners and work through the incident as it progresses. The available assistance or rapid needs assessment team, especially in response, and we've had a few requests for recovery, specific individuals depending on the incident so that a determination can be made as to what resources are actually needed to respond to that particular type of event. A lot of times it is made up of BIA folks, Indian Health Service folks, folks from BIA's Division of Environmental and Cultural Resource Management or the Department of Interior's Office of Environmental Policy Compliments. That would be particularly if we're talking with some cultural issues or cultural items or anywhere else that we need, the U.S. Army Corps of Engineers, USDA, all the partners for that specific event to get the best ground truth, eyes-on assessment of what's needed and what help can be provided. Coordination, this is probably the biggest. We do a lot of coordination and a lot of times it's really making sure that all the appropriate partners that can or want to help are actually informed of the situation. We've had some on several occasions, hey, I'm trying to get a hold of these folks or we're trying to find out who to talk to but we don't know who to engage, we've got help. So that initial coordination with appropriate agencies, again, across the board, federal, state, whatever the case may be, is very, very important, including sometimes, maybe the state and the tribe don't have the best relationship but the state has some resources. Can we help coordinate that on behalf of the tribe at the tribe's request to make sure that if those things are there and available that they're provided? Communication support, we do have multiple communications vehicles throughout the country, fairly robust command post-type platforms that are available during incidents. Law enforcement support obviously is where we're located in the Office of Justice Services. That was a big support element during the Northwest Wildfires with evacuations, preparedness and planning assistance and then response and recovery assistance, as much or as little as we can provide, we hope to provide. We talked about the Rapidities Assessment Team, usually three to five folks. You see some of the disciplines there, public health engineering and, again, natural and cultural historical folks on a regular basis. Almost every incident seems to have a public health component to it in one way, shape or form, so that's why Indian Health Services is one of our most important partners. Capability-based requests. This has evolved very nicely over time, over the last few years. We get requests for things like food or water and we say, okay, we need water and then there'd be this long pause and so now we've got to go through, do you need potable water or non-potable water? Do you need water buffaloes, five-gallon containers, individual bottles, those types of things? So now we try to be very specific to say, hey, what is the capability that you need? What do you actually need done? Then let's take that, go to these other partners and see if they have the ability to be able to provide that capability. So instead of asking, let's example here, hey, we need a bulldozer and a grater, it's easier to say, hey, we need to repair 200 feet of gravel road acting as a dam. Okay, well, depending on the agency or the organization that's helping, they might have an equipment that's closer that can do that, that's not necessarily a bulldozer or a grater, but still be able to meet the need based on what the actual requirement is as opposed to what you think the individual parts and pieces are. Emergency support function 11, we have a very good relationship which is how this partnership got started actually with FAIC and Jessica through the Department of Interest Office of Environmental Policy and Compliance. We frequently provide folks with taskings from ESF 11 through FEMA for which the Office of Voluntary Office of Environmental Policy and Compliance is the lead agency for ESF 11 on the natural cultural historic side. We provided everything from architectural historians to historical architects to other environmental type folks as well as in some cases some emergency management type individuals as well. This is probably our biggest connection during a Stafford type event with respect to natural cultural historic items. So, artifacts and collections and kind of how an emergency manager thinks. The REP program has been very successful for us with respect to taking a deeper look at those things, how important they are, how complicated protecting some of those things can be and resources that are required can sometimes be significant. So, prioritization. Obviously, we want to protect everything. We want to save everything but if that number is going to be low what are the most important things? What are the most important items in a specific building? Whatever the case may be but some type of priority so that if it's really significant a flooding event in particular something that can be seen a little bit down the road as opposed to something like a tornado which is instantaneous for the most part. Can we start to say, hey, let's take care of this first and then we'll work through the rest of the items or the artifacts or the collections after that. And then, are there any special instruction or method? This was a big eye-opener for me when we were at University of Nebraska Omaha. The tribal member who was talking about an artifact, I believe it was a 500-plus-year-old medicine bag that nobody, on their current reservation there's no elder or spiritual person that is able to touch or move that item. And so it has to be a very, very specific person from a very specific line of folks that are educated and go through the list of requirements there and nobody else can move it. And if somebody else tries to move it very bad things are more than likely going to happen to that individual. If the option is can we save it? Do we let it burn up in the fire? Well, we don't want to let it burn up, but you really can't move it. And one of the things they came up with, well, hey, there are some folks in a neighboring tribe that might be able to do that. Maybe we can engage them and see if there's someone more expeditious that could get there if it really did have to be moved in a hurry, those types of things.  They say we got to move it, get it out of the way, and we got to protect it. So some of the very unique things with tribes and tribal communities. And who has been involved in the planning process and or needs to be involved? Planning, planning, planning is critical. You know, if it goes back to the best, not written down, it never really happened. But trying to build into those plans so that everybody knows what is supposed to take place, how it's supposed to take place and then everybody's in agreement. That way there are as few surprises as you can possibly have when an incident occurs. You're never going to be able to predict anything with the type of incident, how long the incident is or anything like that. But the more planning that you do on the front side, it's a lot easier to do to actually execute that plan if you had to. And including as many people as you can. I tell people when we go out and provide technical assistance to tribes for emergency management planning in general, all hazards, type things. Bring everybody, bring anybody that you think might be there. They'll pair themselves down if they really don't have a role with respect to the planning process individually. But a lot of folks get missed, public works. You know, so, and again on the all hazards side, cultural and historic folks sometimes aren't part of the conversation. It seems like environmental always makes it to the table. But are there, you know, the Tribal Historic Preservation Officer or someone else that needs to be part of that planning to even at the larger level to make sure that everybody's aware that, hey, this is going to be part of the process as we move through the potential for response. So then limitations for the protection of collections. Obviously lives are going to go first. Property is going to come second. But who will establish that priority for allocating those resources? And when does that work? How does that work? How do those resources immediately accessible? A lot of times they're not if you're drawing from that third item. You know, if they're going to come from mutual aid or come from an outside jurisdiction, whether it's another tribe, whether it's county, state, a federal partner. Where's it going to come from? How fast can it get there? And if it's a larger incident, are there two of them someplace? Obviously I'm talking best case scenario where everybody works together on everything. You can plan as best you can and try to engage as many folks as possible. And a lot of times we run into situations where they don't provide help or they're not interested, but a lot of times folks haven't been asked and that goes both directions. So again, engage as many partners as you possibly can and make sure everybody understands the importance of the protection of these artifacts, collections, historical items. And these, that's very, very well known, but folks coming from the outside, whether they be federal partners or otherwise, a lot of times they don't understand how important those things are to a tribe, their culture and their heritage. And then kind of leading into that, ensuring responders have an understanding of those unique needs and handling of those type of collections. You know, depending on the neighboring jurisdiction, some of the things, can you think of, you know, is it possible to do an awareness training or provide some type of little pocket guide or other document that says, hey, you know, if we've got folks coming in from the county or the state or from another tribe, wherever the case may be, is there a quick checklist for lack of a better word, a quick guide that says, hey, do this and this and this, but for this, we need someone very specific to work with that individual item or that collection or that artifact. And then, you know, doing that as much as you can, if that's possible, but then also, can we take a tribal responder or just an individual, a cultural historic individual and serve as a cultural liaison to help guide the outside responders. That's been in practice for quite a while with Wildland Fire. When Wildland Fire, especially, a wildland fire, but when you've got four service folks, they'll have folks that will kind of lead the dozer line, if you will, to make sure that, you know, when these dozers are going through cutting lines to help try to prevent wildfires, that they're just not tearing up the sacred sites, whatever the case may be. So, a lot of times, that one individual can provide a lot of benefit working with folks that just don't have a knowledge or awareness of how important those things are, and you can provide them fairly quickly, and most folks are very receptive to understanding that. So, some of our short-term initiatives with respect to FAIC and cultural awareness, cultural resource awareness, our continued outreach to the Museum of Cultural Stewards as part of our program and our preparedness operation, making sure that our rapid needs assessment team plans include someone who has some cultural training and experience. Those folks aren't always easy to find, so we're trying to put together a little roster of folks that could be ready on fairly quick order in a moment to notice, so we'll be able to provide assistance as part of that rapid needs assessment team. And our continued financial support of Museum and Cultural Stewards, management training, which is what this is part of. And these are some of our pipe dreams, everybody's got them. We do not have a disaster relief program in the FEMA so there's not a big pot of money sitting out there waiting for a disaster to happen. If that assistance is provided it comes from another program so roads or whatever the case may be depending on the event and the incident, if there are funds available. A revolving fund for a uniform level assistance and then a structure to be able to do that. This has been talked about several times at the department level also but hasn't, as of yet gained a whole lot of very specific traction. And I believe yes, so there's since contact information as well as mine, those are our cell numbers. Please feel free those are 24-7, we know that comes with the territory so you know, we'd rather get a call in the middle of the night that something's happening than find out that something happened the next day that we think we may have been able to help with. So please feel free to give us a shout, send us an email and then I will turn it back over to Jessica to see if there are any questions comments or anything else we can provide. Back to Jessica. Thursday, we're skipping tomorrow jumping to Thursday. Again at 1pm we'll be joined by Alexandra Allert who actually took part in the rep workshops that Patrick had mentioned this past fall. Alex is a collections care professional and a conservator who is going to talk about how to best relay all of this information to preparedness plans for tribes. So again, thank you to Patrick, thank you to Mike at Learning Times for all of your technical assistance and I look forward to the next webinar in the series. Thanks for attending.


Bonjour Paris, nice to be here, nice to be on this event. So this is the 10th year of Kubernetes and I noticed I already, this is my third talk on a KubeCon and if I count the zero, it's KubeCon which was a Linux-con container-con. And this is what I'm doing normally. My major topic is Kubernetes security and especially in critical infrastructure. So energy, healthcare, German government, but also traffic. I'm a founder of a cloud computing security company. I have several Kovono memberships where I supported even the German IT Planning Council and more or less what I did, I applied a DevSecOps and GitOps strategy which is now being part of the German cloud strategy. My latest customers, but this is not important. Just one disclaimer before this, as a cloud security architect, I've been involved with the two projects I will present later. This is an open desk project by the German government and as a consultant on computer security or Kubernetes security, we in C Lagoon in the aerospace industry. So the history, this is the first thing to notice. There is a promise and I traced it back to 1998, the Linux desktop. This is the year of the Linux desktop. I personally presented Linux in the, I pitched it in the German Bundestag so the German parliament in 2001. This shows me how old I am. And as you might have noticed, it did not really take off. The market share jumped to 4%, which is astonishing and it is more or less that thing which powers Chromebooks. And yeah, but now we have two solutions usable and if I count next cloud and own cloud, also we have four solutions which are more or less ready for production as your desktop in the browser. What are the drivers behind this? So the first driver is security.  The last major breach in Germany took down seven cities. In one of the cities I have been born. So this means with you services, social welfare is under stress. You cannot do relocation notifications anymore and for the business, you have to switch back to import, export notifications on paper. This was the incident in the south of Faesphalia. It's down since November last year. Not fully recovered, 70 cities affected. If you see further the drivers at the moment, we see also that the big cloud vendors, whatever they tell you are not able to guarantee every level of security. It might be okay for your business, for your hotel booking sites, but it is not okay on a governmental or military side. So the last incident is Microsoft and they have been visited by a Russian group. They stole the source code. So if you talk about open source and security, yes, effectively the Microsoft Windows Azure Code now might be open source, at least to the people who want to attack. And what the security community talks about is, you have the four riders of the apocalypse on your desktop, which is Outlook Exchange Active Directory and Office. It is virtually impossible now for smaller teams to make them secure and it is not possible to operate them without stress. And this is part of the movement of Microsoft. They want the users to go into the cloud, but if they themselves are not able to keep the security level high, we have a problem as a government or if you have really secure operations on the way. Personally, it's not only Microsoft. I have seen flaws in every cloud, including Google and Amazon. And this is just a matter of how much you have contact with the cloud. If you look into cloud applications, at least let's say three months, then it's highly probable that you find a flaw which affects the entire cloud. Another driver, how should I frame this, is more or less haircut-driven sovereignty. The US Cloud Act allows the US government to visit all the data, even if it's in European data centers. This is not acceptable. There is a certain fear of the American administration going mad in of the year. And as I told you, there is some doubt in the security of the hyperscalers. They are doing much things better, but on that scale, we really have a problem with our own sovereignty. Problem is that if, let's say, the American NSA goes into the European data, the NSA has also been hacked by the Russians, the Iranian, the North Korean groups and probably by some others. So this means if the NSA can read your data, more or less everybody can read your data. And digital serenity, this is a CNCF survey from 2021. You have to build your own competence. You have to build your own cloud. And this is only possible if you use something like Kubernetes. In former times, it was also OpenStack, but OpenStack was limited to a certain amount of customers. There are a lot of companies in Germany who are able to do Kubernetes on scale. Some of them are here, like Agile Systems or KubaMatic or Server in CloudStack, which I've not seen here. So we have the competence. We have the competence in Germany. We have it in France. We have it in Europe. So why not build our own clouds with our own data? Well, definitely under our control. There's a certain movement at the moment to create a secure stack. And you might have seen all the talks here about the Kubernetes security, edge security, service mesh, minimal containers and so on. What you probably did not notice is that there are also other initiatives from the CPU vendors to create secure enclaves on the CPU and even on the GPU level that you can probably not extract the data even if your workload is running in the cloud. This security community says we are questioning this. You have trusted boot, trusted boot works. Every Chromebook proofs that trusted boot is working. People are telling you that the Linux kernel is secure. You have to trust them. Some people say on our level, the Linux kernel is not secure enough. So they are building a microkernel with automated proofs below the Linux kernel. And then this microkernel is able to be aware of the container workload in the Linux kernel. So the microkernel controls the workload of the Linux kernel and the container workload. This works also with Kubernetes. So they can definitely tell you that it is not only by EBPF, impossible to run an unknown workload. It is also possible by a microkernel. They are in the middle of the proof. So the proof is about 100,000 lines of C++ code. This means they are using automated proving systems and then they can create a microkernel controlling the Linux kernel in a secure way. The other thing is confidential computing. This is also a big buzz at the moment. So you can create a secure version of Kubernetes which relies on the GPU and CPU and clouds. So you have definitely a connection to a CPU. Obviously we need the hardware vendors in this project. And then you are fine. I already mentioned the Kubernetes layer. There's even, there's much more. And then the final frontier more or less is application security. It does not really make sense to run an insecure application on a secure stack. So we need kind of secure workload. In our case, we need kind of complex workload like a desktop. If I ask people what is a desktop, I get different answers. And this is a summary of all the answers I've gotten so far, classic desktop. So yeah, it's a Windows, no, it's a Linux. Which kind of Linux? GNOME, KDE, I personally use I3, but this is a minority. This, it's definitely something with Office. And you can discuss, is it diagram drawing, mail, calender, browser, video conference, chatting, is a wiki part of your desktop? I don't know, project management, some people want that file sharing, ticketing, search, identity and access management, definitely must be involved. You want to control who is chatting with whom. So the desktop suite is not well defined. You have Microsoft Office as a classic idea of a desktop, but you also have the open source tools, which are now integrated into these cloud native desktops. Sometimes it's getting really complicated because some institutions have documents which must be kept for 100 years. This means you need archive systems. If you go to the crypto community, not the Bitcoin community, the old fashioned crypto community who is able to sign documents and is doing the real research, there is no algorithm at the moment, which guarantees for you that the hash will be stable for 100 years. They give you five years, maybe 10. And this means you have challenges on this desktop in certain institutions where you have to re-sign the documents and prove that it has been re-signed before the algorithm is flawed, which you are actually using, which makes it even more complicated. One example from the German government that did a release, another release three days ago, was Open Desk. Here you see what's involved in Open Desk. So you start with an operating system by an invention, which is a Debian flavor. Then you have office system, presentation system. You can use Collabora here. You have something like Excel, table calculation. You have something like NextCloud. You have a project management tools. This is a governmental project. So they need project management. You have a wiki for documentation. You have some simple CripPad for exchanging short messages. You have a whiteboard, which is effectively based on element. You have Open Exchange as a mail server. You have Element itself with a JITC video client. So this is the definition of a desktop as a German government does it at the moment. And here you see we need open standards. We need everything open. It should be part of the German cloud strategy, which means it must be in Kubernetes. It must be independent of the operator. You need some security. You have integration. Finally, not at this release, but in the end release, it must be usable on mobile devices. It must be inclusive so that everybody can use it. It must be a web application. The design must be changeable. It should unite under a single look and feel and it should have something. As a single sign on. So here's a link if you want to it. Unfortunately, the German government insists on having the documents in German first. So if you are German, it's no problem. Otherwise, you will understand the concept and have to translate them into your mother language. This is part of the architectural design on Kubernetes. So this is a typical Kubernetes setup. You have a load balancer at the top. When you have a log in, this is more or less done by an Ingress. And on the Ingress level, you also have something like mod security, which means you have an application gateway inside your Ingress layer. And you have also a packet filter, which is a load balancer. And you have internal packet filters and the security officials, they accept it that we are doing it with some kind of network policies. So the red lines are isolating the application inside the desktop cluster. This means, for example, if something is hacked, if one of the applications is hacked, then you cannot, if the chat is hacked, you cannot go into the documents. It's all on open code. I have been also working pro bono for open code. This is a German GitLab. So this cannot compete with GitLab, with GitHub on size and scaling, but legally, it is now in a control that you have the guarantees that all the applications on open code are complying with German law, which is not the case on GitHub. And this is important for a government that you have, that you imply compliance rule on that level. This is the outcome of a broader strategy, the Deutsche Verwaltungstrategie, so the German administration cloud strategy, which is, this was my little contribution, more or less inspired by the CNCF DevSecOps approach. So we just copied the basic ingredients of the DevSecOps approach into this strategy, and this means it's a GitHub strategy, it has a lot of security built in, and effectively, the base is a DevOps strategy with security built in. This is all published on basis by the Federal Ministry of the Interior and Community in Germany, and this means, yeah, it's more or less public, it's a Creative Commons license, and you can use it. The other desktop, which is already in use by some companies who are doing really security stuff, is VNC Lagoon, which is also a fully integrated desktop with a different focus. I did the Kubernetes adaptation within six weeks, so together with the team, of course, they are using hardened container, they are also using this micro segmentation, network policies, and other things. And the demo, actually, if we look here, the demo is actually this presentation, so I'm presenting out of this cloud, and I can show you how it looks like, so it's a little bit fancy, so they also have an adaptable theme, you can change it by your company. Here you have the file applications, and you have task manager, you have a mail, which is a different mail client, it's Zimbra, and you can send mails to this in the browser, and this is, I think it's a network here. Okay, it tells me I've opened it elsewhere in another window, but this is definitely one mail application, and I've not so many content here, so let's just go further. To compare the both approaches a little bit, it's a network, it's always a network, has a little bit different components, like Zimbra mail and own cloud instead of next cloud, which is not a big difference. They are not using OpenProject, but RedMind for ticketing, they are not using Matrix, but XMPP chat, and the biggest asset is they have a search engine based on Solar, and Solar is a search engine of Bloomberg, who is giving you all the nice content of the American Stock Exchange, and all the financial information you want to have, and one of the less known approaches is they have Cortesar, which is a French company, they have a low-code application, so you can click with very limited knowledge of programming and a fancy application in Cortesar low-code, and they already have a mobile app which is fully adapting to a mobile environment, so you have definitely the goal which OpenDesk is aiming for end of the year, or next year already reached, and they can play with it in a way that they can roll out an application just in a few weeks, which is based on that stack. What are the challenges? The final challenge is always the desktop or the mobile, if you have to bring your own device policy in your company, nobody can help you if the phone or the desktop is not hardened, and so this would be the next step to create some Chromebook-like application for the German administration where you can have a secure version of a real operating system. Another problem always from a security point of view is the browser. It has more code than the operating system, frequent updates, and it is virtually impossible to make security audits on a browser system, and for every application you have to do it on your own, so application by application, and also the interplay of the applications must be audited. Kubernetes services compared to this is quite easy. We have a standard, what we have to do, we need internal isolation, we need minimal containers, we have either some network layer like Calico or some service mesh like Istio or LinkerD, so this is pretty standard. Nowadays this was, in the beginning of the project, it was more challenges. What surprised me is, effectively, mail systems are not cloud-native, and it is really hard to get a mail system which is cloud-native because it is based on POSIX storage. It's not 12-factor, it's not scaling horizontally. It wants vertical scaling, give me more memory, give me more resources, and one other problem is, if you integrate so many components, they have partially their own integration, and you are dismantling the applications and take out the parts which you need for this kind of desktop. Another thing is, in this area of the German administration, we are just starting with microservices, so you cannot think that any microservice language or any cloud-native knowledge is already in the developer community of these services. And same with security, it's not really known how to secure the systems. This was my job, and we made a lot of assumptions which then broke some applications because people, if they start developing microservices, put the entire operating system in a container, and then you have to tell them how to get rid of the components you don't need. From a Kubernetes point of view, yeah, hardened container images is more or less straightforward. I have, on my GitHub repo, I have an example how you harden even an engine X that you can take any engine X with dynamic libraries and throw out everything. So the standard engine X example in Kubernetes is the worst example for beginners of Kubernetes security you can have, but you can do better. Microsegmentation is something you need. Some operations need an air-gap environment. This means you have a difficulty to bring cloud-native applications into an air-gap environment. I also have an example how to do this with a mixture of Kubernetes cluster running only Haber and Argo CD and kind of GITI, small Git installation that you have one cluster per security zone that you can pull it into an air-gap environment through standard tools of Kubernetes. Also a thing is the existing network segmentation. People think we have network segmentation, therefore we are secure. And finally, yes, if you look into it, this is not the case. Another challenge, this was a German army paid a certain version of element that they included MLS, so the message layer security standard and they had their own crypto, which is a national crypto standard. So they needed an Apache license and they are breeding their own version of matrix with the existing crypto algorithms and a top layer with their own standards. So this altogether can give you a high level of isolation and it can be run in secured environments. And it can be used for confidential and secret documents. If you see this side by side, so the left side is open desk and the right side is VNC Lagoon. So you see this is more appealing, this is more neutral, let's to say it. So we will see how both projects will evolve in the future, maybe they will join somehow, maybe because this is definitely so the interoperability of all the components in VNC Lagoon is very high. So this is more or less whatever you want to do on a very administration like desktop, this is more fancy. So, yeah, next things for example, integrate, draw IO, have a full integration layer that you can simply say, fire up a chat with these persons, fire up a video conference with these persons. This will be controlled by an API guy, gateway Angela and what's already going on is integration of immutable archive systems and there is an example where you have the integration of an archive systems into VNC Lagoon where you have one million signed documents per day. This is a bigger system. Challenges, yeah, when Google gave away Kubernetes, this was not a big deal for Google. But if you have all the small and medium companies you have seen here, these are more or less the crown jewels. They have real assets and they have problems if some big companies getting free ride off on that. So all the companies have some hidden secrets where you get a professional version which scales better just to keep the big free riders out. We have seen the same problem with databases in the cloud. So this led to a clash with the licensing. This is this SSPL versus AGPL problem. AGPL should be sufficient because all the big companies hate AGPL. So the first thing you should think about is if you want to negotiate, put your code like Element does under the AGPL then you get the free riders out but they can negotiate if you can make them an offer by contract then you can go back to the Apache license or to another license. What I want to kick off in the near future is to have something like a cloud native desktop foundation under the umbrella of the Linux and the CNCF. We are on common ground, can develop and integrate components in a cloud based desktop and that's basically it and I'm still around until Friday. If you have questions, ask me now. Thank you for coming.


Dave Vellante, I'm with Wikibon.org. I'm here with my colleague, Stu Miniman, and we're live at HP Discover, HP's big customer event. This is theCUBE, SiliconANGLE.tv's continuous coverage. This is our second year at HP Discover, and we're here with David Scott, who runs the storage business for HP. He's the executive vice president of that business line. Former CEO of 3Par, many of you know. The big acquisition a couple years ago that HP made in a bidding war with Dell, HP won. And it's doing great with the company now, but Dave, David, welcome to theCUBE, first of all. It's great to be back. And David has a responsibility for the entire storage portfolio now, not just the single product company that you guys were and tremendous success story there, but it's two years in now. 18 months, not quite two years. Okay, two years in from when we were talking about this at VMworld, right? We were watching the blow by blow. So how's it going? Really well. I think HP's in the process of transforming the storage industry. I think I've mentioned before, we have aggregated some terrific storage assets in platforms like 3Par and StoreOnce, and LeftHand, and Ibrics. And we're really bringing it all together and making a mark on the market. Yeah, I mean, frankly, the portfolio was deficient three or four years ago. I mean, I would go to analyst meetings and I'd say, okay, you got EVA and you got these products, they're good products, but all the cool stuff's happening at 3Par and Compellent and Ibrics and LeftHand, of course, was in accuracy. So conscious strategy. So now the little cool stuff's happening at HP. So this is the thing. You and I have talked about this a number of times, the whole best of breed versus integrated stacks, right? And now the whales are sucking up all the good stuff, right? So what's happening there? I mean, what's your take on all that? Well, you know, I think we feel that the world's been transforming with this evolution towards cloud, the delivery of IT as a service, whether it's private or public clouds, the massive explosion in unstructured data and human information. And fundamentally, all of the kind of leading architectures in the industry by revenue, they were all designed 17, 22 years ago, very different circumstances, not fit for purpose for these new generation of challenges. And the opportunity HP's is seen is fundamentally to bring forward four modern use software storage architectures that are best in breed, comprehensive, going from kind of one end of the spectrum all the way up to massive, highly scalable systems that really deliver tremendous agility, efficiency, simplicity for our customers. Yeah, so let's talk about store once. I mean, you guys are making a lot of noise about that, making a number of claims, very aggressive marketing for HP, going, frankly, going after EMC, talking about better performance, better restore performance. EMC's the market leader, they acquired data domain, Avamar, have a big market share. And now, the interesting thing is store once was developed inside of HP. That's right, it's absolutely organic development out of HP Labs. We have more than 50 patent pending innovations there, brought to market by HP Storage two years ago. Two years ago, we set out this vision of federated de-duplication, the idea of having a single de-duplication algorithm for all of the use cases around the enterprise and therefore being able to pass unrehydrated data through the network very efficiently, rather than all of these incompatible technologies from companies like EMC were deployed in different use cases and you can't pass data between them without going through this massively expensive rehydration cycle, passing it through the network, consuming lots of bandwidth and de-duplicating it again. Very, very inefficient, difficult to manage and we're offering really clean, single architectural approaches and we're doing it with the de-duplication this-to-this backup space with store once, just as we're doing it with three-part on the primary storage space. Yeah, so let's stay on the de-duplication piece a little bit. I mean, the pressure's on, right? I mean, you've got a big market, it's surprised me the size of that appliance market, the de-duplication appliance market, it's now bigger than the tape market was and I thought it would be limited by the tape market. What did I miss? Well, I think- Sorry, sorry, what I mean is, let me be more clear, what I mean is it would be limited by the size of the tape market. It's exceeded that size, why? What mental model did I not get? I think quite frankly, you're seeing the massive explosion in unstructured data. I think by 2020, something like 35 zettabytes of data is going to be kind of stored, 85% of that is going to be human information, unstructured information and it's really been traumatic for legacy backup systems. They just can't keep up with the performance requirements. You have a patchwork of solutions for backup applications quite often, sometimes you have one application for backing up your physical servers and other applications to backup your virtual machines, the third one for cloud-based backup solutions. And what's more, I think a lot of these environments, backup repositories, if you like, are for most customers, they're archive. They don't have separate archives unless there is a legal requirement for them to do so. And so trying to recover information in a granular way from those archives is also a big challenge. And so over the last couple of days between StoreOnce introducing industry-leading performance at 100 terabytes per hour backup speed with our new source-side StoreOnce catalyst software, native backup speed of 40 terabytes per hour, but more importantly, the ability to recover and restore at 40 terabytes per hour, the same restore speed as a backup speed, no compromise, unlike our market-leading competitors. We love when you take shots. Fundamentally, we are servicing this massive explosion in data better, and we're doing some very exciting things with integrating autonomy with our data protector backup software with StoreOnce to allow more granular information access by introducing the world's first meaning-based information protection system, and that's data protector seven, which is shipping now. Yeah, you know, I made some announcements, I guess on Monday, about that integration. So, you know, autonomy's got a big customer base and should be able to move some product through there. The vision, David, of deduplication everywhere with a single algorithm, no rehydration. I love that, by the way. I first heard it in an EMC world three years ago, and then EMC bought data domain and I said, well, that's never going to happen. Then you guys came out with that vision and you started with the vision. You're now starting to deliver on it through things like data protector, and do you feel like customers are going to really hop on that, or do they need education? What are the barriers to them, you know, really adopting that vision? Well, I think the key for us is that there's now a real opportunity to serve as customers better, because, you know, these first generation deduplication technologies, they have no high availability, EMC data domain, you get a node fail, you need to fix it, and then there needs to be manual intervention to restart the backup streams. The B6200 that we introduced in November last year, it has the unique capability, excuse me, of autonomic restart, a high availability solution. The system will detect if there's been a failure and automatically restart the failed backup streams on other nodes within the B6200, and that is the kind of differentiation that we've been offering, higher restore speeds. We're five times faster restoring from the B6200 than EMC is with data domain. Does that matter? Of course it does. How much does downtime cost? 100, you know, millions of dollars a day, and we can literally recover as much in one day as it will take EMC with data domain to recover in one week. So there's a demand out there, and with this new software being supported by backup applications like HP's Data Protector, by Symantec NetBackup with Symantec BackupExecSoon, it's very easy for customers to swap out these kind of generation one de-duplication engines and swap in HP store once technology. Okay, so the pressure's on, right? We should, a year from now when we're sitting here, hopefully we'll be sitting here, we'll look at the IDC data. You would expect that that slice of the HP pie, slice of the backup pie to be much more toward HP's, right? That's the expectation. You guys would be very aggressive about that. Well certainly if you look at our products like StoreOnce and 3PAR, they have incredibly high growth rates. StoreOnce has been running at either near triple digit or triple digit growth rates over all of the last few quarters, just like the 3PAR platform, which has been growing at triple digit growth rates. 3PAR, by the way, happens now to be the largest storage array product line that HP sells, a real milestone that we've passed. Really? Absolutely, but I think, you know, there's incredible momentum behind the business. Even though HP's overall storage business only grew 1% last quarter year over year, under the covers, the more interesting stat was that we grew 8% year over year from an external storage perspective. And when I compare that to the reported growth rates of all of our competitors like EMC, NetApp, it exceeds all of those competitors significantly. So I think you're starting to see signs that there can be positive market shifts. So can I just clarify one of the things you said? You said 3PAR is the leading product, number one product within HP, within HP Storage, right? Within HP Storage, absolutely. Yeah, and so just for people's clarification, I mean these are markets that are growing, you know, the 3PAR storage market, not the 3PAR piece, but the storage market that 3PAR participates in is growing at, I don't know, single digits, you know, 10% right? And the store once market, the deduplication of clients is growing much more quickly, you know, 30, 40%, and you're saying you're growing at triple digits, so you're gaining share there. So, you know, I think back, you know, if he's dialed back 10 years ago, you know, HP dominated the internal DAZ marketplace with server portfolio, and now HP has, you know, a solid portfolio of external storage, but what I find really interesting, and I cover the convergence space, is HP's one of the few vendors out there that can really put those pieces together, the compute and storage. We've seen a blurring of those technologies, HP's partnering with companies like Fusion IO, heavily involved in Flash.  Yeah, well, you know, a whole converged storage strategy is based and predicated on leveraging industry standard platforms, mainly because we're the number one supplier of industry standard servers in the world. And so, platforms like our left hand scale out iSCSI solution directly leverage our ProLiant and our blade system platforms, and that allows us to really deliver complete converged solutions today. And if you look at left hand, we have some really exciting stuff. The whole area of virtual storage appliances is set to start to kind of explode and we've been a leader there. We've had a left hand VSA virtual storage appliance for a significant period of time and in solid state, you know, it was actually the first, one of the first platforms in that space to support solid state disk in industry standard technology. We also, earlier in the year, introduced the P4900 implementation of left hand, which is an appliance solution for scale out. And that was a complete solid state solution as well. So we've made great progress there. We've made great progress in solid state in other areas of the product line like three part. In fact, you know, solid state drives and really team up very nicely with sub-lun storage tiering. Though those two technologies are very co-linked and few people would know but the HP three power platform was actually the first high end platform ever to support the combination of sub-lun storage tiering with our adaptive optimization product with solid state disk. So we've been a leader there, a leader in convergence and we see the opportunity to leverage the fact that we're a big storage player or a big server player and really deliver more value in the conversion industry moving forward. I want to stay on three part for a second because it's such an exciting story. I was walking around the discover show floor and I stopped by the storage area and I saw next to the yellow three part device, the system and then a sign that said, tier one storage for the cloud. So tier one, you were the, you ran the XP product line before you went to three part and that was the sort of classic tier one reliability, availability, service ability. Is the definition of tier one storage changing? No, I think if you look at the environments that the HP three part platform was originally targeted for it was for these large infrastructure as a service for public cloud providers. Utilities. These are effectively utilities. You don't get more tier one than a utility. I still have the red herring all marked up. Exactly and seven out of the top 10 global hosting service providers use three part as their tier one storage for the cloud. And that's fundamentally the success that the platforms had. And as people start to adopt private clouds within their own enterprises, they want to use the same technology that the leading global hosting service providers have had so much success with delivering tremendous mission critical availability over a long period of time. And so that's why in part we're growing at 100% plus growth rates quarter after quarter. And I do want to make a point though. Some people say in HP is three part just replacing EVA and that's absolutely not the case. If you take the example of combining EVA and three part revenues together, last quarter we grew 19% year over year. So the combination of EVA and three part. So we're clearly taking market away from other vendors like EMC, like network appliance, et cetera. I have to, I'm getting the two minutes on here. So I have to ask you, because you are a very successful entrepreneur, CEO of a public company, tremendous exit. And you participated in one of the greatest wealth creations I've ever seen in storage. You timed it just amazing and three part, compelling and on and on and on, I salon, et cetera. Are we going to see a similar trend with the flash guys? The take out of extreme IO at a reported, what was it, 400 and change, even though that wasn't a publicly announced number but that was kind of the whisper number. Are we going to see a similar value creation in your opinion in that flash area or is this a little bit too early? Well, I think it's not clear where the end state value creation in the flash market or the non volatile memory market is going to be. And there are a lot of opportunities for vendors like HP that have very strong server and storage intellectual property and franchises to create value that individual startups may not be able to create by themselves. So it's one of those differences between the opportunities that avail themselves to compelent and free power and I salon, et cetera. These are the situation that the flash market. Because of the end to end systems. Because of the end system linkage that is likely to be a dominant factor in the solid state market. Last question, what's keeping you at HP?  I love the opportunity. I mean, who wouldn't like to take just these incredible assets, drive them kind of hard in the direction of solving real customer problems as they exist today and are going to exist in the future and being able to transform the storage market. And that's what we're doing here at HP Storage. David Scott, thank you very much. We really appreciate all the support you're coming on theCUBE all the time and it's great to see you again. Delighted to be here. Good luck with everything. Thank you so much. All right, this is theCUBE. Keep it right there. We'll be right back with our next guest.


From Santa Clara, California, it's the Cube, covering Technovation World Pitch Summit 2019, brought to you by SiliconANGLE Media. Now, here's Sonya Tagare. Hi, and welcome to the Cube. I'm your host, Sonya Tagare, and we're here at Oracle's Ag News Campus in Santa Clara, California, covering Technovation's World Pitch Summit 2019, a pitch competition in which girls from around the world develop mobile apps in order to create positive change in the world. With us today, we have Team Tech Witches from India with Ananya Grover, Vasudeh Siddhinder, Vanchika Yadav, Aretha, and Anushka Sharma. Welcome to the Cube, and congratulations on being finalists. Thank you. So, your app is called Mate3. Can you tell us more about that? Yeah, our app is an Android application which connects old age homes with orphanages so that senior citizens can spend time with orphan children. And individuals like you and I and everyone else can volunteer and donate to old age homes and orphanages through our app. Okay, and so how would someone use the app? Can you go through step by step? So, as an individual, if you log into the app, you'll see two lists. So if you press on connect, you could either press on connect to an old age home or connect to an orphanage. So for instance, I want to connect to an old age home. I press that button. I would see a list of old age homes. And that list is a list of organizations that have signed up on the app. So I press and they are arranged on the order of proximity to my location. So I press on whichever one I want to view and then I see all of their details. I could contact them, connect to them by messaging them, calling them up, viewing their website or even navigating to their location using Google Maps. Then I could also press on see what donations they require and see exactly what they need. For instance, if an organization requires clothes or blankets, I would be able to view that and donate accordingly. Okay, so it's not just introducing older people to orphans, it's more about like, just like what can we do as a society to like donate to these organizations. And as the administrator of an old age home, you can connect to an orphanage and organize meetings between senior citizens and children so they can spend time together and have fun activities. For instance, we have already conducted two activities in which people mingle together and the children were singing songs and they were even sharing stories. They were dancing together. It was really heartwarming. Wow, and do you have any other success stories from what's happened so far? So the two meetings that have been conducted already were huge successes in our opinion. So we had children from an orphanage called Saib Bal Kripa going to an old age home and they spent time with the elders there, talked to them, mingled with them, learned about their life experiences and in the other meeting children were singing songs and the elders and children were sort of having fun together and it was amazing. So I visited your Instagram page which looks amazing by the way. Thank you. And I noticed that there were a list of activities for them to do when they meet each other so can you tell us more about those activities? Yes, so some of the activities I already talked about them but they could also sort of have a picnic together, get some food for each other or the children could make drawings, write poems for the elders and those can be put up in the old age home itself so that even after the children leave it's like a reminder and a token of love for the elders that reminds them that they are loved and they are cared for and they could also have other activities like yoga sessions together, maybe read books to each other, those are the kinds of activities that are listed on our app. How did you come up with this idea? Well, the initial inspiration or I wouldn't say inspiration, it was a personal incident that happened to me so I lost both of my grandparents within the span of a year. I lost my grandmother to cancer and then after she passed away my grandfather sort of felt like really lonely and that he had lost all purpose in life and I could see firsthand the effects that loneliness could have on someone's mental and social well-being and that led to health problems that he never ever had before and we lost him unfortunately in January this year so I really wanted to work on senior care and then my teammate Vasudha suggested that we should also include orphans within this since we in India especially we have a very strong familial bond between children and grandparents and these children do not have that same connection they grow up without that kind of nurturing love that only parents or grandparents can provide to a child and so we felt that we should be solving both of these problems and bridging them together in some way and that's how we came up with this idea. Oh, it sounds like it's an amazing idea and it also sounds like it could help more than just the community that you're in but it could really help the whole world because so many people deal with loneliness especially in old age or orphans who want love from a grandparent or a parent so it just sounds like it could be really useful to a lot of people. Thank you. So how has your experience at Technovation been? It has been an amazing journey we weren't expecting to make it to this level and it's just phenomenal and a huge honour to be standing here and we began by just we just formed we are all classmates and we have known each other for a while so we just found out about this competition formed a team, started working towards it and we were really passionate about this cause so working and taking time out of school to work on this just came naturally simply because how important this cause was for us personally and then so throughout the Technovation journey we kept on learning new things not all of us were aware of how to code applications so we used Technovation's curriculum and other online resources to teach ourselves those skills along the way and developed our entire project submission and yeah it's just amazing to be here now preaching in front of such a huge audience. Absolutely, it's such a huge success for all of you so can you tell us more about how you all met? You said you're classmates right? Yeah, so the four of us are classmates and Anushka is in another class within the school so we already sort of knew each other and I found out about the competition online and Technovation seemed really like a great platform especially because it's like it specifically wants to empower women to code and go into still male dominated fields like computer science and entrepreneurship so the four of us are science students and she studies commerce and that's how we came together and started working. That's amazing so where do you hope to see this app you know if you get the funding if you win today where do you hope to see it in five years? Right now our app has already been launched in the Play Store and we have about 1000 plus downloads in the future we would like to not just expand out of the Delhi and Sia region where we're currently working and go pan India and hopefully in the future in about 10 years maybe spread this throughout the world because I feel like communities across the world can benefit from Maithri. That's amazing, best of luck to all of you and good luck for your pitch tonight and congratulations once again. Thank you so much. So this has been Team Tech Witches from India you're watching theCUBE, stay tuned for more.


What do people mean when they use the terms transformation and disruption? Should I be concerned if I'm hearing them at my organization? And how is the conversation about transformation relevant to me? Great question. I would also start with saying I'd probably separate transformation from disruption. I think people's, the term disruption has been used a lot, right? And everybody, it's kind of got this cool coinage to it. Like, I wanna be a disruptor, Uber disrupted an industry. But you can't really plan for disruption. It's really not a strategy.  I made a change within an environment or an organizational structure or even an industry. So I'd probably put that to the side and say transformation is what we'd focus on. I don't think you should be concerned. We hear a lot about change. I look at transformation as organized change rather than chaos. Being able to take control and say, where we're at today, the status quo, the evolution of the world around me, the positioning of my organization, the systems and the people and processes within it, it all has to change. And I wanna manage it, I wanna transform. Transformation, you know, the caterpillar to a butterfly. It's a big change, but it is organized and something really good comes out the other end when you can guide it that way.


Hello, everyone. This is Vikas and Nick from TATRADE. At TATRADE, we work at Istio and other service mesh technologies around Istio. In this talk, we are going to discuss and share our experiences and learnings that we gained while working with some of our large enterprise customers, trying to help them in machifying their multi cluster installations. This talk is roughly divided into four sections. In the first section, we will discuss about the complexity and the manageability issues in the large scale multi cluster installations. Then, assuming that service mesh and Istio is going to help in managing these problems, we will see the gap and pain points if we try to follow the Istio community suggested multi cluster installation approaches for overlaying Istio mesh on the multi cluster installations. And then we will discuss in the next two sections our approach, which is a bit more practical and how it is addressing the pain points and the gaps that we found in the community suggested installation approaches. So next passing over to Nick to start with the problems of the multi cluster installations. Thank you. Congratulations, everybody. You've made it to multi cluster. You've successfully scaled your application from a single cluster to multiple. But when you're looking back and your SLAs haven't improved, you broke a bunch of security rules in doing so so that you could get the networking to connect properly, and then you realize that management has got a lot more difficult. It seems like the problems that you're facing are bigger than the ones you were doing earlier. So trading off a better SLA for more problems didn't really seem advantageous to you. And so we're going to walk through some of these problems that a lot of our customers have early on when they're adopting the multi cluster model. The first one that we want to talk about is this is a typical deployment that you would see when you go from a one to multi cluster environment. Typically, you would have a low balancer within the cluster. Engine X ingress is the default one that most of you are probably familiar with. And then we connect all of those clusters and do the routing with a tier one, which we call tier one low balancer, which is typically your cloud low balancer. And that's so you have a single point of entry and can out between multiple clusters. But your team one, you've been tasked to move your microservice from one cluster into a multi cluster environment now. You're going to follow the coattails of team two, who's already been doing it for a little while. And so you're going to take your application and stick it behind that same default low balancer that team one or team two has been using. But all of a sudden, you start to know you have more outages than you have before. You've bound yourself now to the failures of team two. Team two's errors have now become your problem. Because when they have problems, they're low balancers. They cause the low balancer to be removed from the pool, which is now causing outages for you. And so there's cases where the going multi cluster doesn't actually make anything better. And so we're going to have some solutions that will address these problems in a very practical way. Secondly, when you go to this typical multi cluster routing, you don't have a good way of being able to tell your microservice how to locally address services if they're within your cluster or within your region. And so to get the HA availability, you're going to have to call outside of your cluster to that tier one low balancer to get access to that other microservice or service that you're trying to access.  But if you route locally and that pod or that service is down, you have no failover. And finally, as your multi cluster architecture grows, you can get some really complex dependency chains. It'd be very difficult for you to trace all of the consumers that depend on you and your functionality. And so without ServiceMesh, you'll find these problems are very prevalent. And so then you'll be looking for that solution, which ServiceMesh seems to be the one that will solve these problems for you. And so as you're adopting ServiceMesh, there's a lot of questions that our customers are asking. And how do you go about doing it in the correct way? Am I adding more complexity to my architecture when multi clusters already made it complex? I chose Istio. And now when I upgraded it, I lost connectivity to everything. And so we're having larger production outages. Do I need to re-architect my multi clusters setup so that I can adopt ServiceMesh? And so a lot of these questions we'll answer later on the sites. And now I'll pass it over to Vikas to start with the Istio solution. Sure. So yeah, there is no denying that Istio is super powerful. It can, Istio can help in managing these complexities of multi cluster installations. But how exactly I'm going to overlay Istio mesh for multi cluster installations? Because there are different teams, different personas, different administrative boundaries over these clusters. So how one is going to do that? So we'll start with taking a look over what the Istio community documentation has been suggesting. So at the moment there are in the latest release, which is 1.7, there are two multi cluster installation approaches. The one is replicated control planes and another one is shared control plane. Because of the resiliency requirements, we can rule out the shared control plane approach. And let's take a deeper look at the replicated control planes approach. So how this works is that in your clusters, each cluster has an independent Istio control plane running. And the CA of each of the Istio's control planes on these clusters is configured with the intermediate CAs which are generated from the same shared root CA. The services which are the shared services and are supposed to be accessed from the remote clusters are exposed through service entries. For example, here the service foo, which is in the cluster 2, is exposed on the cluster 1 using a service entry. And the host name in the service entry has this dot global prefix added to the name and name space of the service. And the endpoints of the service entry points to the gateways of the remote clusters. At runtime, it looks something like this. When a client makes a request, it goes to Kubernetes, a DNS server, Kubernetes, Kubernetes DNS, we have to run with a core DNS plugin, which is shipped by Istio. So all the dot global prefixed host name queries, Kubernetes DNS, pass these queries to core DNS. And core DNS results these host name DNS queries with the virtual IP of the service entry.  And there the listener is pointing to the remote gateway where the actual service is, the backend service is running. And this way the request reaches the cluster 2's gateway. And SNI host still has the host name which has dot global prefix, so it won't make to the actual backend service. So there an on-way filter is configured to translate this dot global from the SNI host to the dot service dot cluster dot global. So now this gateway and the cluster 2 can finally forward the request to the actual service implementation. But can we use this in our production environments? Unfortunately not. And why not is because it doesn't support local development of a routing out of the box. We cannot simply, if we have a local instance of the service running in cluster, ideally it is desired that the first priority should be that the request should be served by the local instance. So we should be able to somehow the local instance should be part of the load balancer pool. But we cannot simply here in this approach, we cannot simply put the cluster IP in the service entry endpoints. And the reason is this dot global thing. We can fix this. We can make it correct by adding virtual service to rewrite this host name, converting this dot global to canonical name of the service in the local cluster. But this problem is not the only problem. There are much bigger problems than this in this approach. So the host names in the clients are directly depending on the actual backend implementation of the services. So let me explain you with this example here. The actual service is foo, which is running in the foo and has name space in the cluster too. So the clients on the remote clusters here at the cluster one should use the host name, which is the service name, actual service name dot service name space dot global. And this is pretty bad. Because if there are n number of clusters serving this API, serving this service, then the owner of these services cannot change the backend implementation of the service at all because the clients will get broken in that case. So in other words, we can say there is no abstraction. The clients are directly depending on the actual backend implementation. And there is no very straightforward locality of a routing as well. These points are already acknowledged by the Istio community as well. And we're also part of this community. And we have been working to come up with better installation approaches. So in the one dot eight, we have this a new approach. And this replicated control plane is not there anymore in the coming releases. But so in this approach, for example, which is coming, here also we have a still control plane running in each of the clusters. And what change here is this, that the IstioD of each cluster is watching the API servers of all the remote clusters. So this solves the locality aware routing problem because it has all the local endpoint and the remote endpoint in the same LB pool. But the problem of the direct dependency on the canonical name of the service and namespace is still there. Still there is no abstraction. And plus these scalability issues are there and the security concerns are there. Because these clusters are owned by different teams and they may not want to expose all their internal implementation of the services to other teams. So that's where we had to come up with a different approach to meet the requirements of our customers. And now I'm passing over to Nick to explain our approach. Yeah, so we took a look at this and looked at our customers' problems. And our customers have many, many clusters that they want to connect together and span multiple clouds. And so we wanted an approach that was somewhat simplistic and practical to their use cases. And it didn't require a lot of architectural, core architectural changes within their environments to adopt this service mesh architecture. And then we also wanted to put a little bit of the application developer in mind when we deploy these. And so what kind of developer mindset did we try to assume when we were coming up with a solution? Well, developers just wanted to consume applications as SaaS products, even internally. They wanted to access your API, but assume that it's HA and that you're routing locally or efficiently. And so they don't really necessarily care where your service is hosted. It's that they can just reach it easily. They want to spend more time on implementing the features that are really going to drive that product, rather than spending time on a lot of the implementation details with auth, routing, networking, and a lot of stuff that comes with service mesh or even without. And they want to be able to advertise their own products effectively and easily to either external customers or to other teams within their organization. And so with that in mind, we came up with a much more simplistic approach in Istio installation. And that's deploying Istio essentially isolated per cluster. We deploy the control plane on every cluster. We scope that control plane to only know about services within the cluster that it's residing. And so we have a locally scoped mesh. We say that you should manage these control planes externally. And you can do so with a number of tools, CI tools, GitHub. And so that'll take away the managing everything individually. And then the final point is we really want you to embrace gate-base a little bit differently than you are today and expand upon the use of those. So why do we want to do separate Istio control planes? Well, we really want to align the failure domains of your application to that cluster. And so if you push bad configs or you upgrade Istio incorrectly, and it causes an outage, it is now localized to that cluster that you were working. And so it gives you a lot more control over not bringing down your entire environment. And it aligns that Istio control plane with the underlying cluster that it's running out to networking and node management and stuff like that. And so it's a lot more effective for our customers to manage them these way, especially for outages. And then it also allows you to do safer upgrades. So as you upgrade Istio, you might be able to pick a cluster that has less usage. Update that one. If it goes successfully, then you can roll it out to other clusters. And this would differ from the shared control plane that was one of the Istio recommendations. But we have a problem. We can't. The applications cannot see services in other clusters. And so we need to adjust this problem. And so the way that we touch rate want you to go about doing this is by embracing gateways. And gateways are just low balancers that are fronting your applications. But we want you to use them on a product-focused architecture. And so currently, you're probably using the NGINX in Grass Gateway. Or if you're using Istio, the defaulting Grass Gateway in Istio system. But we want you to get rid of those and move to this product-focused gateways. And so in Istio, that's really easy. You can just stand up any number of ingress gateways and then determine the services that are behind it, or upstream from it. And so these product-focused gateways then allow you to control the microservices behind it as if there are one API. And so you can expose your API internally or externally via this gateway. And now you've aligned the failure domain of this gateway with the product that it's supporting. And so you won't have these shared gateway problem where other products could bring you down. And it just requires you to stand up more gateways. And then you'll be able to talk across cluster a lot easier and we'll explain why in a second. Then you can also push your authentication and authorization circuit breaking up into this gateway and then tune it specifically for your applications that are behind it. And so it's a really purpose-built gateway for your product. So in a cluster, this is kind of what you would imagine logically what is happening from using a gateway. The consumer namespace on the right here is not related to payment API product, but if those consume that API. And so logically you should be consuming it at the gateway level. That allows you to scale microservices, add functionality behind it without interruption of that service. And so when we go to a multi-cluster environment, this becomes really easy. We can then just replicate that payments namespace into another cluster with that gateway as a whole package. And then the consumer namespace now has multiple endpoints to reach you at. And so we add those hosts to that consumer namespace and now it has two options. So making your payments API highly available. But we didn't have to stop there. We can do some more intelligent routing that's saying if your consumer is in the same cluster as the gateway that it's consuming and that gateway is closer than say an external gateway in another cluster, let's prefer routing locally first. And so if you're in the same cluster we can actually use the sidecar to act as the gateway. And so in this example, this is what we're doing. We're using the sidecar to act as the gateway to those microservices to access them directly. But in the case of failure or outage we can actually reroute requests over to the cluster two in the US West region. So you still get that highly available but now we've added a component of this intelligent local aware routing. This is really a cost effective for our larger clients that have high volumes of traffic and egress and they're paying a lot of for egress data. So why should you embrace gateways more than you are today? It gives you a way to abstract your APIs in your microservices and with a product spin not a product focus. It's really architected for growing those in a multi-cluster environment. It's more of a copy paste than a one-off or trying to figure out who owns the local egress gateway so you can attach your resources to it. And then it aligns a lot more with the non-mesh architectures that exist today. They're very low balance or centric and so gateway really fits that low balance. But we're just empowering the gateway to be a lot more intelligent about the traffic that it's routing. So if we put this all together and what we're doing at Tetray is we're making that gateway discovery automated. And so from any number of clusters that you can stand up the gateways will automatically be discoverable and all other clusters. That means we don't need to know about all the other services, microservices running in other clusters. We just need to know where those gateway ingress gateways are. We are making sure that when you route to those other gateways outside of your own cluster that it's encrypted and it's authorized. And then we're also implementing locality-based routing. So to improve your cost savings and then failover to failover effectively. And then finally we're using a newer technology to SEO which is mesh DNS, but it allows you to eliminate the need for that namespace routing that Vikas was talking about earlier. You can essentially use your own abstracted DNS for these gateways to more represent the product that you're offering. So hand over Vikas to talk a little bit more about how we're doing some of this routing and gateway management. Thanks, Nick. All right, so from the service owner point of view, exposing a service to the external user has the same security concerns if the intention is to expose the service within the mesh but to the remote clusters which are sitting across the public internet. So essentially whether it, so even the request is coming from within the mesh but it is coming through the public internet. So with this point, we in our model, the service owner is supposed to just expose the service to the external world over the chosen gateway port and the user is supposed to configure whatever security measures, authentication and authorization, he or she feels comfortable with to deal with the public internet security concerns. And what we do programmatically is that the gateway hosts, the APIs which are exposed to the external world, we automatically expose these APIs within the mesh to be consumed from the remote clusters. So what we do basically is that this 15443 is a reserved port for the Istio MTLS. We discover what the APIs are exposed to the external world and we expose the same APIs over Istio MTLS on this reserved port for the East West traffic. And additionally, in addition to the Istio MTLS, we apply all the authentication and authorization configurations which are with the user has configured for the external world traffic. We apply those on this 15443 port as well. So it is like double secure. In addition to that, now this is a very important slide. We create a local service entry as well. And the point to focus here is that the host name in the service, for example here pay.example.com is same as the host which is exposed to be consumed by external world over 9443 port. What this means is that whether the client of the service is within this same cluster or in some other remote cluster or some external user, all are consuming this service with the same abstracted API pay.example.com. And the remote instances and the local instance are part of the same load balancer pool. So, and just one more implementation detail here. To achieve this, the endpoint in this service entry is the local Kubernetes cluster IP. And now this is how it will look if there are more than one clusters where this backend service has instances running. So, this service entry in the local cluster has two endpoints. The one is the cluster IP of the local service instance and this another one in the green card. This is the gateway IP address of the remote cluster where one more instance of the service is running. Now we are going to see at runtime when the request flows from the client, how load balancing happens and how it reaches the destination service. So, first of all, when the client makes a request, the client makes a request to the abstracted API pay.example.com, whether it is for the local instance. So, the client need not bother about where the service instance is running locally or globally or wherever it is. So, abstracted API is used by the clients. This goes to the sidecar and in the sidecar we are using a feature where the proxy hijacks the DNS queries and it caches the DNS and resolves the DNS query with the virtual IP of the service entry. And then after the DNS query gets resolved, then the request starts from the client container with the virtual IP of the service entry and now in the sidecar matching with the virtual IP, the listener is picked up. The listener has the endpoints to the local instance as well as to the remote instances and because of the locality of your routing, in ideal case, the local instances are picked up and if there is no local instance or the local instance is in failure mode, then sidecar will route the request to the remote gateways and when it reaches the gateway, the request is over the STMTLS plus additionally, the gateway will authorize the request using the extra authentication and authorization configurations which the user service owner has configured for the external world. So here it can be any external oath service which will authorize the request in addition to STMTLS and if everything passes, then the request gets routed to the actual instance. So just to summarize here, here we are using a abstracted API. So there is no dependency directly on the backend service and secondly, what API is a service owner wants to expose he has full control on the exposure of the APIs. For example, like whatever API he wants to expose to the consumed within the mesh or to the external world, the service owner will expose this on the gateway. It is not that the whole cluster is being washed upon by the remote clusters, remote histories. So because of the time constraint, we could cover this much only, though we are doing much more interesting stuff, we are doing multi-tier, we are handling multi-tier architectures as well. We are handling where the clusters are in the, the clusters are part of the different VPCs which are not directly connected but talking through a shared VPCs. So we would really like to explain all these things but because of time constraints, we cannot talk about this. So we can take these things offline if you are interested in knowing more about this. And I think that's pretty much all. Thank you. Thank you.