All right, I hear we're good to start. Hello, everyone. Thanks for lending us your time and joining us on this talk on Zonal Outage Operational Stories. My name is Shyam Jheerikunta, and I'm joined by Jyoti Mahapatra. Both of us are software engineers from Amazon EKS at AWS and come from a team that ensures reliability of clusters during zonal outages or partial failures of various kinds. Together, our team has successfully mitigated dozens of such events, learned some valuable lessons along the way, and used those to engineer highly available and reliable Kubernetes clusters. And that's exactly what we're going to talk today. Our focus will be on the Kubernetes control plane, but some of the learnings also apply more broadly to user applications or components within the data plane of the Kubernetes cluster. So I'd like to start with this model. It's called a Swiss cheese model. I wish I could say French cheese, but I guess Swiss cheese just has more holes in it. So this is a model that's used to analyze and manage risks in large, complex systems with many moving parts. It's an effective mental model that's been used time and again in various industries like aviation, health care, nuclear power plants, and even distributed software systems. Essentially, each hole in the cheese represents a weakness or a dormant failure that could manifest itself in the presence of an external trigger or hazard. For example, it could be a bug in the flight's menu-resist system that gets triggered at a specific airspeed or altitude. If you're familiar with the whole Concorde fiasco, you probably know what I'm talking about. There we go, a French reference. OK, so diving a little deeper. As software engineers, we often see ourselves striving to build systems or services that operate reliably always, but the reality is distributed systems are practically impossible to be made flawless. Your block of cheese will have holes in it, and in the fullness of time and scale, there will be different kinds of failures that will happen in individual components, microservices, and even their interactions. For instance, hardware that your Kubernetes cluster runs on will eventually fail, and this includes compute, your CPUs, GPUs, and even volumes and disk storage. Even the best health checks sometimes they don't detect certain failures. Node sometimes get partitioned over network. In weird ways, sometimes it's partitioned bi-directionally, sometimes only in one direction. So the best bet is actually to try as much as possible to decouple these failure modes, make them uncorrelated, and thereby reducing the overall single points of failures that your system has. The idea is that by reducing such single points of failures, you are increasing the number of things that need to go down in your system before it breaks down, which basically steeply, even exponentially reduces the probability of that happening. And that's exactly what this Swiss cheese model view is showing here. There are a few ways to interpret it. The most common one is each slice of the cheese can be thought of as different components or layers or layers of defenses in your system that have been built to protect it from, let's say, downtime. And each layer, of course, has some sort of holes in it at different places and of different magnitudes. And the red lines that go through these are hazards which the system is facing. And the one red line that actually made it all the way till the end that was able to successfully wreak havoc because it had something good going on for it. That is, all the holes in its path have lined up and making it a single point of failure for the system. So avoiding these sort of failures needs deeper thought and rigorous engineering. We're going to talk about that a little bit going forward. But first, I also want to present another concept of redundancy. So this is a foundational mechanism that a lot of distributed systems leverage to achieve high availability. Redundancy is essentially the concept of running multiple replicas of your software and hardware so that if one or more replicas fail, the others can continue to keep the lights on. Similarly, in databases land, high durability is achieved by maintaining redundant copies of the data across multiple hosts. Redundancy helps reduce the probability of the system failing bird in the face of uncorrelated independent failures, not correlated failures. So let's say there is a bug in the component that you've replicated in a way that, let's say, a bad right to a database has crashed all three replicas, then it is a correlated failure. And redundancy in that case doesn't help, actually. So to avoid these sort of all-for-one-one-for-all situation, let's talk more about how can we uncorrelated these failures. And I want to introduce at this point the concept of a failure domain. So if you're familiar, commercial data centers lay out the hardware and the underconnecting network infrastructure over a specific topology. And this topology is typically hierarchical and a simple version of that is what's shown here. It's basically a tree where servers and the other hardware like disks are arranged into rows or racks and which kind of roll up into maybe rooms and that rolls up into your entire data center. And each rack is typically connected with L2 switches. It's commonly called leaf level switches. And they, in turn, are connected to larger groups of servers through L2 or L3, spine level switches. As you can imagine, there are failure points at each level. For instance, a single server in a given rack could become inoperable due to a hardware malfunction. Or a broken ethernet cable or the switch could take down the whole rack. And similarly, a power loss could take down a whole room of servers. So there are layers of failure domains here. Then which one do we actually use for redundancy? As it turns out, the most common practice, especially with some of the larger cloud providers or data centers, is to use a single data center as the unit of redundancy. It's essentially a physical building or a bunch of co-located buildings housing the hardware and other supporting infrastructure like power supply, cooling systems, disaster recovery mechanism, and it's typically called a zone or an availability zone. Now a region is essentially a collection, a geographical area a cloud provider may operate in, which is a collection of multiple of these data centers or zones. They're often placed in close proximity to allow for low latency communication between those agencies, typically connected physically. Through massive backbone cables. And while they are kept close enough in terms of proximity, they also need to be kept far enough so that the chances of being affected, multiple of them being affected at the same time due to the same hazard or a force of nature like a flood or a power loss should be low. So this idea is essentially the idea is that when a zone is down, the rest of the region continues to operate smoothly. All right, so with that in mind with an understanding of redundancy and failure domains, let's apply it to the Kubernetes control plan. This diagram may be a bit oversimplified, but it shows a typical highly available architecture where all the key control bin components like HCDE API server and core Kubernetes controllers, they run on a set of three instances that are spread across three different AZs. And even the load balancer that actually takes north-south API traffic from clients is replicated across the AZs. Similarly, the in cluster API traffic that comes from clients via the Kubernetes service IP that is also backed by API server endpoints from multiple zones. Now let's go back to the earlier discussion around single points of failures. So with redundancy, we might think we are at a better place, especially given these sort of components have a failover mechanism inbuilt into them. Like for instance, HCDE, it's a quorum and leader-based system. If you're familiar with it, it needs two out of three replicas to be running at any given time to be operational, so you can lose one replica. And among the other two replicas that are still available, which form the quorum, you need to have one of them as the leader, an active leader at all times. Similarly, controllers and Kubernetes work based on a lease-based leadership concept, where if one replica is unable, ideally the other, that one should fail health checks, and another replica should just take over. Similarly, with API server, let's say a single API server is unable, the load balancer, it should stop receiving traffic from the load balancer, typically because of the readyZ, liveZ health checks that are configured on the load balancer. So it sounds simple, right? It feels like we kind of have what we want in terms of redundancy and high availability. So if AZ1 completely goes down, say we have all these mechanisms that should allow us to smoothly fail over to AZ2 and AZ3, except zonal failures are not always hard failures, where everything in that zone just comes down to a grinding halt with a failure rate of 100%. That's just not the reality all the time. Sure, there are failures like fiber cards and hurricanes and complete power laws that leave a zone completely unreachable, but zonal failures can actually virtually take unlimited forms in terms of size and shape, and most of the ones we've actually seen operating Kubernetes clusters are gray failures. So power or thermal issues that affect individual rooms in a zone or networking problems between zones that cause some sort of elevated latencies or spiky throughput behavior and so on. Also zonal failures aren't always just caused due to lower level physical stuff, but it can also be due to, if you're familiar with zonal infrastructure services like block storage or L4, L7 load balancers, these are services that are designed zonally and the services that actually operate these components, they sometimes also deploy software in a zonal fashion just to reduce the blast radius of bad software bugs in a region, which means that if a service that you depend on has done a bad deployment in a zone that could cause an outage or some sort of impact for you. So yeah, I think the bottom line here really is zones may completely fail, they may totally become unreachable or partially in terms of slow requests, occasional timeouts, et cetera, and basically everywhere in between. And to be able to respond to such events as cluster operators, we believe at Amazon EKS and more generally within AWS that we need to be prepared with deterministic mechanisms to respond to such non-deterministic events, which means despite the nuances of a specific failure mode or a potentially limited understanding of the failure modes emits the event while there's already a burning fire, we still want to be able to guarantee a very clear positive outcome, which in the case of the cumulus control plane means the API has continued availability, and there are no, by continued availability it's not just uptime or healthy, it's things like you don't see API errors, you don't see high latencies and stuff, that and also for all the various controller workflows they should be able to operate without any increase in their downtime. So essentially we want to be prepared for these sort of non-deterministic failures with a strategy where we can, without even sometimes fully understanding the event, safely way away from a bad zone into a healthy zones. With that I'll pass on the mic to Jyothi to talk about the fun part around the case studies we've seen. Thank you, Shyam. Hi, everyone. I'll walk us through some interesting use cases and case studies that we have seen while operating this. We learned that zonal failures are not deterministic and failure modes depend heavily on the nature of correlated failures. When correlated failures happen in a single zone Kubernetes components fail very differently based on their retry strategy, circuit breaking, health checking, and any other failure honey technique. The failure modes that we'll talk about today are for API server, HCD, controller manager, cloud controller manager, and these are just different components. API server is stateless, HCD has quorum, the controller manager's work based on leadership where even if there are more than one replica, one is doing the work really, and that makes the system very complex. The components itself do not have a notion of zone built into them. It is up to the platform administrators who is putting the components to work to understand the nuances of zones, zonal failures, and how it may impact the components. These depend on the administrator to be able to pull a card to say, I know something is happening in a particular zone, can we not use this and remove the chance of correlated failures? Let's look at the picture and see what's happening here. So in this diagram, there are three zones, and in this diagram, zone one is partition. This partition is caused by a slow volume attached to the HCD node. Again here, the network is not fully gone. The instance or the host are not fully gone. Just that the volumes attached to the host are responding slowly. When that happens, the general latency of the system increases. And any calls or requests to that particular HCD node in zone one has a high latency. Now HCD has peer health checks. Other HCD nodes are health checking on this HCD node in zone one to say, how are you? And it is just taking a long time to respond because of the latency. Eventually, the latency may increase so much that the health checks fail. When the health checks fail, this HCD node in zone one loses quantum. It thinks it is the only node in the system. Now coming back to the API server part, we use client-side load balancing on API servers.   So an API server here, in this case zone two, which is healthy really, has a connection to an HCD host in zone one. This means that a problem in zone one is already leaking out into zone two. It causes non-determinism in the system. In this case, the HCD is not terminating the watches that it already has. Any new traffic to that HCD node will not go through because reads, writes, and watches, they need quorum while creating the connection. But once the connection is created, it does not terminate them when the leadership loss happens. In this situation, because the connection never terminated, API server cache got stale. It never got an update, even though other HCD nodes are progressing. In this case, again, we had a leader in kube controller manager, also in zone two, because KCM is watching an API server to give it a node or whatever resource you're looking for, it's not getting it. In the meantime, let's say node controller has a problem, it stops working. And any other new nodes joining the cluster at this time is just not joining the cluster or doing taints and tolerations. Nothing is working. So what do we do with it? But let's talk about why did this happen. This happened due to a co-related failure. In our system, we have reconcilers that look at health of the HCD nodes and terminate them when it thinks that this node is not healthy. We just terminate that and let a new instance come up and reconcile itself. But this reconciler itself also was having a zonal failure. This correlation caused the reconciler not to be able to terminate the host in zone one, the HCD host. So the fix was simple in our case, because we handled the intra, we made sure that the reconciler also terminated the HCD process first, followed by the host termination. Now, even if the host termination does not go through fine because of its retrying or it's just taking a lot of time, because the HCD process is gone, the connections are severed and any new connections are going to the healthy zones. What could we do next? We could go to the HCD node and make changes where the server always terminates connections when the leadership is lost. We are also thinking that we could do more client-side stuff with GRPC discovery service. GRPC has this concept of Envoy discovery service. We could use that to say if a certain host is experiencing higher latency or higher error rates, don't use that for load balancing. We could use GRPC health checks to influence the traffic routing so that the API server does not use the HCD in zone 1 in this case. A lot of issue links are here. We are working on them, so we'll probably arrive at something. Let's look at scenario 2. In this picture, again, zone 1 is partitioned, but it's not fully partitioned, just that the API server is experiencing higher than expected latency. The users now are not having a consistent experience. Sometimes they take a quick, but sometimes they take long. And in this case, this particular API server in zone 1, any traffic coming to it through the load balancer or through the cluster IP is experiencing a very heavy latency. Now, to make this experience consistent for users, we employ a mechanism called WayAway. We deal with the load balancer in a particular way and the cluster IP in a different way. For load balancer, we use a concept called Application Recovery Controller. It's a public API. And for all clusters in a region, we go and tell the load balancer that, hey, don't advertise this particular IP in zone 1. Due to that, any new connections coming to the load balancer does not use zone 1 anymore until we ask it again back to. The cluster IP is tricky because Kubernetes handles this part. The API server has an advertised address and an endpoint reconciler. The endpoint reconciler takes an advertised address and makes endpoint objects of Kubernetes. And they don't fail health checks, even if the API server is unhealthy. They are like DNS. Qproxy uses the cluster IP to round robin load balance between the API servers based on these IPs. We made a change such that if we tell through the fleet that zone is experiencing a problem, then the API server is now zoneally aware. We made it so. And with that, the endpoint reconciler stops advertising the IP in zone 1. With that, any part of the system which is depending on cluster IP or endpoints, they directly go to the other API servers. They don't go to zone 1 anymore. Now, an important nuance to that is the considerable overprovisioning. Let's say we take one API server out. That means we are reducing the APF quotas of the system by a certain proportion. And most times, this just walks out because we overprovision the API server more than what it would need. So we look at metrics, throttling metrics and we have certain laddering. So with certain requests, we set certain quotas. But when this kind of zone affiliates happen, most probably customers, other customers are also seeing that, which means worker nodes are shifting traffic, pods are moving away and evicting. So it pounds the API server with way many requests at this time. So we have also a system where we track the throttling parameters or metrics in the API server, and if we see that the throttling is occurring way more, we go and scale it out. We create more replicas of the API server to withstand the problem, so that users don't experience a problem. Let's look at the third one. In this case, we have a reconciler that does a health check throughout the fleet. And the health check is if X number of health checks fail in Y amount of time, terminate an instance. Maybe something is wrong with the instance and let the system reconcile back. In this case, when zonal failures happen, let's say the health checker is fine, the API server is also just doing fine, but the network stack in between is just behaving abnormal, then the health checks will fail. We observe that in such cases, a large number of clusters just lose one API server because the health checker terminated them. It's not bad, it's not good especially for large clusters because API servers has a ton of watch cache and KMS encryption done, and it's just not good for a brief interruption to kill all of that and start a new API server again. Most possibly, during these correlated failures, we don't want to try and put a lot of load on dependencies, in this case, or KMS services, or other dependent APIs because they may fail and eventually back off. When they back off, recovery takes even longer. In this case, what we do is we circuit break our health checker so that when you see that certain threshold of health checks are failing, it slows down and if it is even more, it just stops and pages us. And interesting aspect is that we track a lot of zonal metrics and this health checker is health checking all the API servers and pushing a lot of metrics to our backend systems. We consume those metrics and create a system that can detect and tell us that one zone is really having a certain problem and we use the same thing for zone throttling all these parameters and feed into the system and then the system tells us and pages are saying, I think something is going on, you should go check. And it automatically starts the way shift process. The way shift is such a safe process that you can just deal with that and then an on call can go check, release it or keep it going. The next one is the cloud controller manager. In this example, again, zone 1 is partition, but it's partition in a way that just the DNS resolution is failing in zone 1. When that happens, most components don't fail except anything that depends on remote APIs or internet. In this case, cloud controller manager. Because the cloud controller manager is able to still get access to the cloud controller manager. So the cloud controller manager is able to get access to the cloud controller manager. It's not failing health checks. Many components today, upstream and our own components, they can't detect such nuanced DNS failure from remote APIs and feed into health check. And as a platform owner, there are like 10 different components of the cloud controller manager . There is something where we added zonal awareness to the controllers. And when we understand from the previous slide, I mentioned that we can detect zonal failures, when we understand that something is happening zonally, we go again push an API which instructs all of these controllers to relinquish these deterministically. And with that, no cluster will see any downtime at all. Because this is so undeterministic because DNS is cached in most cases. So even if DNS is out, it's okay. These controllers will keep working unless some controller restarts, host restarts and now DNS is not available anymore. It's so undeterministic. Let's look at how it looks like. So in this picture, we can see that we have a lot of information across zones. And this is from one of our test simulations where we let the system fail and our zonal shift kicks in and the leadership moves away. So the blue line goes up because it is now taking all the leadership and the other lines, I think the orange, if there's no replica available really to take leadership, don't take it. Don't do this. Okay. This is the last one. So the next one is HDD durability. During HDD upgrades, we do a rolling update. We have three HDD nodes at steady state and we kill one node, let it come back to the cluster, kill the second, let it happen repeatedly. Now, if a correlated zonal failure happens at this time and host has been terminated and yet to come back up, the system is vulnerable. It is available. It can still serving quorum, but if any other node has a problem, it loses quorum immediately and that's a risk. Most of these implementations out there take a backup of the HDD periodically and then apply them when quorum is lost. Although it has a availability risk where from the time when the quorum is lost until the time the restore happens, the system has an availability loss, no restore rights can happen, but the most riskiest part of it is the ability to run from that time, say 15 minute, 5 minute, whatever the periodic interval is, we lost the data. So we improve the durability posted to the system by doing a static volume approach. So we have static membership and static volume. All the HDD data is on the disk and those disks are never thrown away and this is what we do. They just come back up and start right there. There's no chance of any data loss. What is the chance that this may happen? So in our workflows, the most critical part is now a new host came up. We detach the volume from the old one and attach to the new one and assign it the membership it needed. There's a slim chance that something may go wrong at that time. So we can do this again by saying we won't let HDD updates go through when we know that a zonal problem is happening. With that, we have a couple of infrastructure such that if we issue an API server update, it will go through without updating HDD. We can put that in a queue and when we know that the system has recovered, we can go look at what is going on. What is going on in HDD? Anything that bad happening in HDD should not leak out to the customers and we try to ensure that. These are slack threads, slack handles. If you have suggestions, feedback or anything that you want to brainstorm, plenty of links. All of them are spread out across the slides. They're all gathered here. I will be working with upstream to suggest things that we see and make the system better.


Hey, what's up coach? Welcome back to my channel now today. This is a very short video I want to talk about something that I see a lot of coaches Do on social media and this is a big no-no That I recommend you either stop doing or if you see someone do this you should stay far away from them All right, and here's what it is. It's called fake scarcity and What I mean by this this is coaches who will post things like hey, we have a clinic that we're doing. We're only trying to get 10 People signed up and then on the day of the clinic. There's like 30 people there or They say there's a discount only offered for this many people and then they give everybody the discount now if you operate your business that way what happens is over a longer period of time people stop believing you and they don't take action on the Promotions or the marketing that you send out because they know that they can just get The discount or whatever it is you're offering whenever they want because you either extend it or you let everybody do it Now the way that scarcity actually works is by telling people the truth and by sticking to your guns And let me give you an example a couple weeks ago. I promoted a live class This is for coaches that I'm connected with on Instagram. This is a four week promotional marketing class that Where it's very simple. It was like we have a specific discount that ended on a specific day and The people that reached out to me After we did that they were like, hey, I want to do this. Can you still give them the discount? I just said no and Think about it. If I said yes How disrespectful is that to those who did take advantage of the discount and the time frame that I said they needed to Sign up by right so that that's one of those things if you have any sort of scarcity in your marketing or any sort of deadline you have to stick with it and Don't lie about it because when you lie about it people start to lose trust with you when people start to lose trust with you Then they're gonna stop working with you. All right, and that's why I see, you know There are coaches that do this sort of stuff That lie in the marketing all the time and they do it because they want to make more money You can't have a successful business when you're lying to people So that's it for today's video. I hope this helps and if you want to have like real scarcity Then tell the truth and stick with it. All right. See you later


The bicycle and pedestrian counts we do every year, and they help us to prioritize areas where, you know, infrastructure is needed for pedestrian and bicycle safety, as well as just help us to get a better grasp on, you know, the public is definitely necessary for these counts. We need volunteers, as many volunteers as we can get. They can sign up on our website, columbiasc.gov, and the counts begin on September 7th, and we're going to be counting over 30 locations in the Columbia Point, a few downtown, as well as places out in Harveston, out in five points downtown by Richland Library. We're trying to get a very comprehensive view of what's going on in the city.


Happy Sabbath sense, happy day. I am glad to be here to praise the Lord this Sabbath day. He has been good to us. He has been our God. He has walked with us. And we are glad that He always blesses us as His children. Thank you for coming. And I would want also to tell you that today our services are going on early because we will also be able to join the great evangelism campaign that is going on in town. We will do it online. So all of you are welcome. We have a series that is going to run after this one. So all of you are welcome. I am informed that there are some who are going for baptism at New Life Church. And they have ended up here. Please you are requested that you can be given a direction. You just walk outside here using this door so that you can get the direction so that you be baptized there. Those who are coming from other churches and they were going for baptism, please be directed. Thank you so much. Welcome. Our theme for this day, our title is saying, seen us only hope. Meaning that a sinner would have no hope in no more circumstances. A sinner is hopeless. There is only one chance of hope. A sinner have only one chance. Only one way that he can be able to have hope. And this is through the cross of Jesus Christ. This is what I call the sinner's only hope. And I invite you that we may be together in this as we go through a chapter of Psalm 130. Psalm 130, we will go through the old chapter. And those who have their Bibles in their hands, you can be able to open it for one reason that there is something that you cannot get online. There is a title there written in my Bible. It is written a song of ascents in the titles. Some of your titles may have written it song of degrees. Some other versions could have written pilgrim songs or a pilgrim song. There are many songs. There are 15 in number pilgrim songs or song of ascents. There are 15 in number. So just one of them you will find it written a song of degree or a song of ascents or pilgrim song. There are 15 in number. They are starting from Psalm 120 to Psalm 134. I love these songs because always the Jews used to sing these songs. As they were going for three major festivals which they used to attend in Jerusalem. They were going there as they were singing the 15 songs. Maybe one of the times they would be singing Psalm 130 as they are marching to their festival. And these songs always they used to sing them as they were going for the festival after the captivity in Babylon. 7 years after the 7 years in Babylon they would be now going to worship in Jerusalem as they sing these pilgrim songs on their way. Some of the songs they would be singing as they start the journey. Some of the songs they would be singing as they enter the city. And some of the songs they sing as they are on the way leading to Jerusalem. When you look at all those songs you will get a feeling that this song is somebody who is starting the journey. Setting off. And some of the songs you will find them. People were settled on worshiping. This song 130 starts by saying, Out of the depths I have cried to you O Lord. Lord hear my voice. Let me read up to us for fast before I come back. Let your ears be attentive to the voice of my supplications. If you all Lord should mark iniquities. All Lord. Who could stand? But there is forgiveness with you that you may be feared. Let us pause from there and start again from verse 1. We will proceed and finish. From verse 1 still it says, Out of the depths I have cried to you O Lord. This man is somebody who is somewhere in the deep. Somebody in deep trouble and seeing no way out. And saying, when I was seeing no way out I cried unto you. The time I am feeling that I am in a financial crisis I will cry unto you. The time I would feel that I have a family crisis I will cry unto you. Out of the depths I cried unto the Lord. Meaning that this man knows that however deep the problem might be. The prayer here in God will be able to hear that prayer. One of the depths that we find with the Samistir is that he is in the depth of sin. And he is saying, however deep I might be in sin. The Lord will be able to hear me. Then he says, Lord hear my Christ. Let your ears be attentive to the voice of my supplications. This man is saying, I know you will hear. You might be from heaven above and I am deep somewhere. Still I know you will hear. Open your ears to hear me. Then he says in verse 3 that if you all Lord should mark iniquities who could stand. If God would be able to take all my sins and put them in remembrance. Could I be able to stand before him? If God was interested in punishing me for my sins. Do you think I could be able to stand? Definitely I could not be able to stand. Israelites could be asking as they are going to worship in Jerusalem. As they sing this song that Lord it is because of our sins. That you banished us into the land of Babylon for 70 years. But even in those 70 years we have done nothing. Nothing good that we can bring as a plea before you. It is only because of your goodness that you have brought us back to the land of Jerusalem. Lord if you were to mark the sins that we always do. Do you think we can be able to come and worship you? No we cannot be able to come and worship you. This man is like what Paul is saying in Romans 7. I feel the test is coming to be the same. Romans 7 from verse 14 through 17. It is like the same test that Paul is writing to us. When he says that I just want to quote it and come back. When Paul is saying that we know that the Lord is spiritual. But I am cannot sold under sin. For what I am doing I do not understand. For what I will to do that I do not practice.   But I find that thing which I hate that is what I do.  But I find myself just breaking the Sabbath. I find myself hating my appearance. How I appear, how I appear. That doesn't show that I am a Christian. The way I dress. But I find myself doing this. That is what Paul is saying. Then he says in verse 16 that if then I do what I will not. I will not to do. I agree with the law that it is good. But now it is no longer I who do it. But sin that dwells in me that does it. I am a sinner naturally and nothing I can do about it. That is why the Samist is saying if you were to count my sins. Could I be able to stand? Then I remember this Samist also David is crying in another Psalm 51. David is crying for his sins in Psalm 51. Psalm 51 and Psalm 130 are among the seven penitent Psalms. Penitennial Psalms. Psalms for asking for forgiveness. And seeing that we are unable to do anything because we are sinners by nature. David after committing adultery and killing the husband of this woman. He comes to the Lord in Psalm 51 verse 3 and says. I acknowledge my transgressions and my sin is always before me. David is saying I know I acknowledge my transgressions. When I am walking I am always feeling like I am an adulterer. When I am walking I am always feeling that I am a murderer. My sins are always before me. When will I forget these things? And then he cries to the Lord in that manna and in verse same chapter verse 11. He cries to the Lord and says. Do not cast me away from your presence. And do not take your Holy Spirit from me. There is sometimes you have gone too deep into sin. And you feel very down. You feel very rejected with God. Until you feel that you are far from the Lord. Then David is telling the Lord. That do not take your Holy Spirit from me. I am feeling your Holy Spirit is leaving me. Do not take him away from me. Then he says in the next verse. Restore to me the joy of your salvation. And uphold me by your generous spirit. Those who have left sinning there is a joy that you always enjoy. There is something in your life that is always very enjoyable. This is what David is saying. Restore to me the joy of your salvation. Those who are saved are relieved from several burdens. And is crying Lord. I want to remain in your salvation. My sins are always before me. But I need this joy of salvation. Then in the next verse he says. If you do all that and restore me into salvation. Verse 14 verse 13. Then he says. Then I will teach the transgressions you awaits. And sinners shall be converted. Lord. The sins that have committed are so grave. But if you can restore me. If you can confirm to me that you still love me. Then I will testify that you can forgive those who committed adultery. I will testify that you can forgive murderers. If you do that to me. I will confirm to them that you do it. I will confirm to them that you are a good God. A forgiving God. God who accepts his children. And when I will tell them. Because they have seen me do it. And you are forgiving me. They have seen us. They will be converted. That is why the sum is 133 and 4. In 133 and 4 he says. That if you Lord should mark iniquities. Oh Lord. Who can stand? Do you know the reason why he is saying this? That you Lord. If you want to make sure that every sin is punished. From everyone. Do you think there is a human being who could stand? The Bible is saying. That all have sinned. And fall short of the glory of God. That is Romans 323 again. The Bible is saying. That none is righteous. Not even one. None is righteous. Not even one. That is why the sum is asking. Lord if you were to mark my sins. Could I be able to stand? Let's come closer. Please look at yourself.  Do you think if the sins that you always commit. If they were to put plain before us. Do you think you would be able to appear in the church. And do you think it is very pleasing to God. To appear before him with those sins. It is not pleasing to me. Because sin is hateful to God. Then he says in verse 4. Sorry before I go to verse 4. There is a verse in Jeremiah 13 verse 23. Jeremiah 13 verse 23. Maybe I paraphrase it says. That an Ethiopian change. His skin. Can a leopard change his fort? Can a black man change his color to be white? Is it possible? Can a leopard change the fort in its skin? It is not possible. Man cannot change his character. Man cannot change his condition. We are sinners born in sin. And it is sin in you. That always makes you commit sin. But. Okay, I can see that verse. Can the Ethiopian change his skin. Or the leopard his fort. Then you may also do good. Who are accustomed to do evil.  You are accustomed to do evil. And you cannot change. Then this is why the Samist is crying and saying in verse 130 saying. If the Lord should mark my inequities. Or if you all should mark my inequities. If the Lord who could stand. Then verse four is saying. But there is forgiveness with you. That you may be feared. The Samist is saying. That Lord you forgive us.  And in you naturally there is forgiveness. That you may be feared. The one who will be forgiven. Is the one who will see the depth of sin. Like David. And run before the Lord that my sins are ever before me. Because you cannot change yourself. The Lord will be able to give you his spirit. That can be able to transform your life.       But he loves you. That is why he provides a solution. Jesus Christ to pay for your sins. By the way after accepting him. He justifies you.   Saying that you are righteous. Then he covers you with the garb of righteousness. The cloth of Jesus Christ. And that is justification. By the way that is your title to go to heaven. You have done nothing right. You have only asked for forgiveness. And you are entitled to go to heaven. And then when he entitles you to go to heaven. He works with you through the power of the Holy Spirit. To change you from sin to righteousness. That now as you walk. You find yourself character is changing. It is you. It is not you who is doing good. But it is Christ in you who is doing the good that you are doing. That is why we are saying it is Christ's righteousness. Even when you are doing good. It is Christ doing in you. And I love it when in verse 4 some is saying. That Lord, I thank you that there is forgiveness in you. You are partner in the family may not forgive you. Your children may not forgive you. Your parents may not forgive you. But in the Lord there is forgiveness. There is somebody who might have refused to forgive you. But with the Lord there is forgiveness. Why don't you be confident to go to the Lord and tell the Lord that I am a sinner. Lord forgive me. With you there is forgiveness. I cannot change myself. You only need to forgive me.  Then he is saying the reason why God is bringing to us salvation that sheep is because it is for the reason that he may be feared. For the reason that we may know that he is God who can do all things in our lives. Then let me finish the other rest of the verses. Verse 5 is saying. I wait for the Lord. My soul waits. And in this word I do hope. This means to say. With all my problems, even my sin problems, I will wait for the Lord. And I will hope in His word. Meaning that I will hope in His promises. Then he says. My soul waits for the Lord. More than those who watch for the morning. Yes, more than those who watch for the morning. I think you have seen the poster. Sometimes I love communication team. They do imagine things and they do them very well. The poster that they put there that says. Sin us only hope, that poster. You can see a sin us only hope is it was darkness. Somebody is waiting for something, some light to come in the morning. Somebody is saying I am waiting unto the Lord. Like the one who watches for the morning. Look for a watchman who watches through all the night. This man is eagerly waiting for the horizon to change the color. To see that morning is coming. They start waiting as the sun sets in the evening till dawn in the morning. They are eagerly waiting for the light of the morning. Then this man is saying I will hope in the Lord. With all the things that I have. I have nothing I can present to the Lord. I will wait for His promises. It might be dark. Darkness might be long. But I will continue hoping till it will dawn. Then he says. Oh Israel, verse 7. Hope in the Lord. For with the Lord there is mercy. And with him is abandon redemption. And he shall redeem Israel from all his inequities. They are saying as Israel, as the nation. We can have no hope except in the Lord. If the Lord was to punish us according to our sins. We will not be able to stand. But with him there is forgiveness. I just want to invite you that you know your sins. I want to present to you that with the Lord there is forgiveness. You know how far you always move away from the love of God. But God is pleading with you that you may come closer. He has provided all the means. Your only hope is at the cross of Jesus Christ. Your only hope is the grace of the Lord. The Lord is gracious. Somebody is telling me that I have minasena. And I want the Lord to forgive me my sins.


Hi, guys. Hi there. How are you doing? Good. Welcome to everyone. Yeah, we've got a doc at the moment. I'm just going to post it in the chat and also in the Slack panels. Docs. If anybody's got any thoughts or questions as we're going along, then maybe pop them into the docs, the Google Docs. And if we don't have immediate answers, then we can follow up. We haven't got much of an agenda today. I personally was trying to get something to show. I may try to show a live demo or something, but I'm not brave enough. We'll see if we have anything. We've gotten really bad about putting people on the spot and giving them no heads up beforehand, James, don't you? I've learned from the best world. I thought a good topic of discussion, actually, would be for Ian to give us a summary of the latest BDD test. Because that's pretty awesome, actually. Cool. OK. Yeah, I can do that. Yeah, on the spot. No warning whatsoever on the fly. No safety net. Yeah, so I guess that's pretty much what those tests are really for, really. A safety net. So there were a few things about the existing tests, which were kind of like I found hard to work with. Like one of the things was the vendering, like the vendering of the dependencies actually for the harness itself was kind of hard to work with in terms of trying to fit, because we're sort of dog-fooding and we're using a bunch of the commands from within JX itself in the harness in order to like pipe the logs and these kind of things. Yeah, so someone's dog is not happy about this at all. Yeah, so yeah, they were kind of out of lockstep with the actual main project itself. So one of the things I wanted to do was like vendor the dependencies in a way so that they matched basically whatever was in like currently a master in the actual JX project itself and then use those dependencies, use those methods actually in the test times. The other thing that I think is like a big thing will probably become even bigger as we add more and more tests is a whole sort of parallel execution of the test suite because running that many sort of quick starts sequentially just takes forever to run basically. So I kind of got around that before with the old test harness by kind of rolling in parallel, which is like a little binary that you can use to execute. But you don't really get any control over streaming output of logs or like all of the log output will be interleaved so you'll see kind of the output on sort of eight different things and at the same time really difficult with debugging or that kind of stuff. So switch over to Djinko. Djinko basically handles the whole sort of parallelism, it's hard to say, thing by effectively compiling. It kind of splits up the tests at runtime and then kind of compiles them into their own binaries and then executes them from like a sort of centralized place using like a sort of status server that gets spun up as well to actually record the output. And that gives you lots of cool options like only output the log if the test actually fails so you're not just seeing reams and reams of sort of logs for things that are just passing normally. You only actually see the log output if that execution comes back as failure. But I have like an internally inside the library there's a thing called gexec which is basically just a wrapper around exec that enables you to kind of control execution from external commands and also write expectations specifically against those commands. So you can say like I'm expecting this process signal or I want to wait this long specifically before failing this test or whatever. So there's kind of a lot more control especially for sort of our use case where we're just looking to sort of execute sort of binaries and stuff. And you're looking through and you've also added something into to generate that living documentation which then we can publish on to the website as well. Yes, I mean the other thing that's quite cool is there's that I have a sort of an abstraction called like a reporter and obviously like the basic default test suite has a default reporter but you can write your own as well and then you can kind of make it do whatever you want. Another really cool thing about that is that they have hooks that enable you to synchronize all of your parallel tests into one thing at the end when you're finished. So you're executing your tests across eight different nodes eight different tests running sequentially but it will basically wait for everything to finish compile the results and then you've got access to this sort of reporter structure that enables you to create whatever you want. So I mean there's a little, but under the hood it's a little bit fiddly. So what I'm actually doing is basically marshalling every nodes sort of test results into sort of JSON writing that out to the file system and then reading those all back in compiling it together and then writing it out again as HTML. But you can see the B output, JSON output if you wanted sort of like database reporting or whatever format you want really. And I've just hooked that up with JX docs. So it just spits out it's actually an MD, MD file. But that accepts valid HTML as well. So it's just got sort of the time that that build completed and then sort of programmatic listing of all of the sort of top level description blocks. So you're describing Jax Quickstart whatever or just describing a certain feature. It's just gonna output all of those into that. I had to break out my slightly rusty I have to say CSS skills to actually update the style sheet for the docs. So hopefully it's okay. I mean it's pretty sort of neat and tidy and stuff. And I've just added in like a little SVG tick or cross depending on if the suite passes or fails. And I updated this morning as well just to say that basically anything that we mark as pending although that will come back as pending in the actual test suite run in the doc it will just come back and say that that's a failure which I guess is, I mean I could amend it so it just says that it's pending in the report as well but I can't figure that's a good way of sort of encouraging us to complete our pending specs. Well, why are you talking? Would you mind if I shared my screen so I could show people what this project is? Even though I put it in the chat because this is recorded if someone's looking at the recording they won't have the recording of a chat I don't believe. Is that right? I'm sorry? Yeah. Yeah, isn't it? I've pasted the link to the project it's under the Jenkins X organization. I've pasted that in chat but if you're watching this in a recording you won't have access to the chat. So just look at my screen I can do that just so people can see where it actually is. Is that opening? Taking a long time. Now I can't do that, it's taking too long. Anyway, the name of the project is BDD-JX. Yeah, BDD-JX. It should be a lot easier as well if anyone looking to get involved in the project wants to run the tests hopefully and stuff. There's really only one thing that you need to have set which is just your gig organization basically your gig username in an environment variable and at the start of all of the test runs it will actually spit out a little thing saying what you've got that set to and what you need to set. So the intention is there, I guess, to encourage people to try it for themselves and report things if they've got problems or just generally help with the contribution of getting things really solid and working really nicely. So we're going to run these tests every time we do a change to the platform. Yeah, so I think they run on every PR now which is really cool. It's just like especially running in parallel I think we're going to run into some kind of underlying issues that maybe we wouldn't have run into before which is why I was thinking about the whole mitigations for merge conflicts which I think when in this morning I think James Lionel Messi James merged that this morning so that's my unofficial name for in order to differentiate the two James's. We have Lionel Messi James and other James. I probably need a better like thing than other James. Ronaldo James? Ronaldo James, yeah. It's a fight to the death of the Ballon d'Or. Cool. Yeah, so I think that's pretty much where it's at. So just for anyone that wasn't in our conversation this morning basically sometimes the issue can be if you kick off a bunch of different app creations at the same time you can get merge conflicts into the requirements Yammel because all of the PRs are hitting at the same time and basically it's the same three lines that it's trying to write against. It doesn't know what to do, you get conflict. So a mitigation for that is basically to reorder the requirements by name. The intention being that the insurgents will come at different line numbers and therefore you won't get as many conflicts. So it's definitely possible to still create conflicts. I don't think there's any way around that and there's some really nice code that does like rebasing that seems to work really well. So it's just like an extra kind of little added addition on top of that really to try and make it even more kind of smart about how it does it. And just to get somewhere with that. It's been great because you found some great bugs that we just wouldn't be exposed to. So having those being run automatically on every pull request has been superb and then the parallelism as well because that's what people are going to be wanting to do part of that build, right? Yeah, that's the thing. Yeah, I think it's really cool. That's the thing, it is such a complex system and it handles so much for you already. There's kind of inevitable that there are going to be these edge faces and there are going to be these little things. Yeah, I mean the intention is just to try and sort of close those gaps and in particular make it really, really easy for people to get started. Because I think if you can kind of get beyond, if you don't have any kind of technical issues in that very first kind of starting up process then the main sort of feeling you get from looking at the project is kind of like, wow, this is kind of amazing. It's doing all this stuff for me. Which if you're kind of really excited about it and you come to it and the first thing you see is some sort of error or you get blocked in some way. Kind of a bit deflating. So yeah, it's really just kind of exposing people to the awesomeness of Jenkins X without like, you know, teething trouble. It's awesome. I wonder if we can get the BDD test to label the quick starts if they don't work. And then we could filter them out from the JX quick start command. So anything we know doesn't work, we could just hide it basically from the... Yeah, like automatic... Yeah, that's really cool. It's a bit of a rubbish filter, but if you put a prefix of WIP, then it does that right now. But we could come up with something better. Yeah, like maybe we could just label the GitHub repository or something. I'm not sure if there's an API to do that, but something like that would be awesome if we had some way of... Just some canonical place we could say these don't work. Yeah, nice. Then to also have maybe an option on the CLI to actually then to include the beaters or the other quick starts. You can actually select them if you... We only have the box, the stable ones. I think that's like the key to it as well, is to really... There's obviously a bunch of domain-specific experts that we're talking about Elixir this morning. Haven't had an enormous amount of exposure to Elixir myself. I think it's really cool, but I've only just played around with it. There are probably people using it in very production grade Elixir environment that have got a bunch of best practices about how to bootstrap an Elixir app and encouraging contributions so that we can use best... In that specific domain, we can use the best practices for this specific thing without having to be experts in everything ourselves and making it really easy for people to contribute in that way. I think that would be really awesome and we'll get a much broader range and the quality will probably be a bit higher than if we're... Each of us trying to figure out like how on Earth do... How on Earth is... What is the best practices for an Elixir project without looking at it specifically? I wouldn't know.  That's great. Ultimately, it won't be good. Longer term is these quick starts actually rather than just a basic page of Hello World, it'd be nice to actually offer some links to training or various information so that people can try out different languages, have that automated CI-CD pipeline set up and then maybe links to wizards or something, how they can start building on top. It's also like a training involving new languages. Yeah, I mean, it'd be great.  For instance, Rust and played around with Rust a little bit, but we've got a great quick start for Rust. It's like, boom, there you go. You've got a Rust app up and running. It's live. You can query it. You can play around with it. You can put it through that whole process straight away. And it's very difficult to... I can't really think of any other way of getting from sort of nothing to... This is being delivered in various environments. You've got previews, you've got everything. And you get all these logs, centralized metrics and everything as well as all other bugs. So it's a really good forum for extending skill bases as well. If you've got other things that you want to play around with, we've got a quick start for it. It's a great way to get involved in that too. And people can contribute because I think a lot of those quick starts, people have been contributing. If people do have a quick start, then we can just forward that into the repo and we can tie that into our automated tests. So that's kind of another way that if people, from the organization or from their team, they have put together a boilerplate quick start that uses the same typical way they package or build their applications. If we have a quick start, then that gets tied into our BDD tests that get run on every single PR. Yeah, that would be an amazing way of involving people as well. Awesome. Thanks, Ed. I was going to say, some other folks have been doing some interesting things. I know James has been loving the time that he's been spending with EKS. But we've always got on EKS, right? Yeah, Amazon is awesome. Yeah, I've definitely been spoiled using GKE, which is so easy to get stuff done. And Amazon's powerful, but different. Yeah, slightly harder to use. But yeah, we've got EKS is just about working. I haven't really blogged about it yet because it's not 100%. We've got JX. Let me show you the CLI. Let me try to share my screen. Is this my second one? Yeah. I'll keep talking in there. Oh, there's no player. He was taking me out trying to share my screen before. Let me try. Does that work? Oh, there you go. I'll turn off my video just in case that helps. The internet isn't that good. So we have a command line now for JX create cluster, EKS create cluster, there we go. Create cluster. And we have an EKS command. Yay. And this actually works. And under the covers it's using EKS CTL from the Weave folks. CLI. Because it turns out even though EKS is a thing, making an EKS cluster is still really quite hard. And so if you use this, so basically create cluster EKS downloads EKS CTL if you haven't already downloaded it and it installs it in your path and all that kind of stuff. Then it uses that to spin up the EKS cluster. Then it installs JX on top. And it's surprisingly complicated to create an EKS cluster. There's all kinds of different things. There's a VPC, there's a service role, there's a node policy thing. Anyways, a node pool and then you create the EKS cluster. So there's all kinds of different Amazon resources. That all appear in CloudFormation. So another covers EKS CTL is using CloudFormation which is kind of cool. So it's kind of working. And by the way it uses ECR by default. So I've switched Amazon and EKS providers to use ECR by default. The elastic container registry which is cool. The one weird thing is whenever you create a project on Amazon, you can't just push to an arbitrary Docker image to ECR. You have to create an ECR registry repository, I think it's called, for every Docker name you make. So whenever you do JX import or JX create spring or JX create quick start, I've hacked that that if your cluster is, if you're using ECR as your Docker registry, then we do the AWS create ECR thingy. So we kind of try and hide the manual stuff you have to do with Amazon a bit. So right now if you use Amazon with COPs or EKS, it uses ECR which is cool. So we don't need to use the local Docker daemon or mess around with insecure Docker registries or that kind of rubbish. So that's kind of nice. The only thing that's missing which I've been fighting with now for a few days is by default when you install Jenkins X, we install an Ingress and then we find the host name of the Ingress and we use a load balancer like a single load balancer for the cluster and then we generate Ingresses to all the different services and all the different new spaces using that Ingress controller. The current issue, I'll stop sharing my screen now actually and put video on instead, the only issue is when you're using ELB, Amazon's Elastic Load Balancer, there's no single canonical IP address. Each zone gets its own IP address internally so there's three IP addresses and those IP addresses change all the time every day. So once you create your cluster, the next day disappears again, which is not brilliant. So you can manually set up Route 53 yourself by hand but that's kind of painful. So I'm just hacking expose controller so that we can use path-based... In fact, let me share my screen again. I'll just start with a kind of half demo. Let me put my screen back and hide Rob's face. If I open the window, I'm using an Amazon account right now, an Amazon cluster, JXCTX. Yeah, I'm using a COPS cluster right this second. So this is COPS. If I do JX open... Where did my window go? There we go. JX open. Notice the path-based ingress. So we're using a single domain and then we're using paths afterwards. So now we're using paths. It means rather than using the IP address, we could use the ELB host name. The only problem is using path-based breaks everything. So if I open Nexus, Nexus doesn't work and you can't look into Jenkins because Jenkins doesn't know it's at least path. So now I need to go through all of our charts and inject the path into all the charts. Oh, it's horrible. But yes, which is really unfortunate but I can't really see another way forward. I don't think you can... Sorry, I was going to jump in on the whole path thing. You know you're using an ingress control already, right? Which is basically just nginx. Yes. So not that it's going to be like unicorns and rainbows but you could potentially map paths using nginx at ingress to domains, like sub-domains. Is there a way of hiding the path to the... Yeah, that's what I mean. You basically rewrite all of...  How do you do that? That's awesome. You've got to do it for Monocular at the moment. Yeah, so... Because it's just nginx, if we come up with some sort of fancy custom nginx conflict to the ingress, then we could potentially rewrite all of the paths just into sub-domains and then you wouldn't have to just kind of handle it there rather than getting involved in... Oh, that would be awesome. Yeah, let's do that. That sounds awesome. Thank you. I'll just drop this into the Public Slack channel in a second, but we do something similar for Monocular to rewrites. We've got two back-end services and Kubernetes. Yeah. We add annotations onto the service and then ExposedController generates the ingress rule with those nginx rewrites annotation. Yes, that's awesome. Okay, I'm going to try that. As soon as this call is finished, I'm going to try it. Yes, I was scared by having to go around and fix every application that's broken. That sounds a much better idea. Thank you, Ian. That's awesome. I owe you lots of bidders. I'll try that after this call. That's brilliant. So yes, so Amazon's close. Like EKS and COPS is close to being awesome. So we can switch back to using the ELB hostname for Amazon once this path stuff works, which would be awesome. Yeah. Very good. Is there any questions or anybody got anything more? I can give an update of what I've been messing around with, but maybe it's not very interesting compared to somebody else. I was just going to mention one other thing. I've finally documented how to change the configuration of Jenkins X, of course, in this Slack. I put it on the chat room earlier today, but the other thing is we've now got, you can specify the Docker registry now as you're installing by a dash, dash Docker registry, which we documented on there. So just if you don't want to do the MyValues dance, you can specify it on the command line as well. And GCR should work. There's just an IAM thing we need to figure out to let the build pods push to GCR. I haven't figured out the magic yet to get that to work, but hopefully that should be fairly straightforward, and then we can automate that in JX install or JX pick custom GKE. It's been nice to always use GCR by default on Google. I wonder if the scaffold folks might know something about that. You would hope that scaffold is pushing to GCR. Yeah, that's a good call. Yeah. Nice. Before you say anything today, I was just shaking down a very trivial bug which has caused me a bit of heartache today. That's HALM. So I'll be using an old version of HALM. It's 28.x and we're now on 29.1. The JX version, when you pick that out, it just tells you the client and the server both the same version. It's just clearly wrong. I've only just discovered. I'll try and find out why that is happening. It does mean if your client and your server are different versions and your builds will fail. Yeah. Yeah, we should make JX install or JX pick list to check the version of HALM. Also maybe upgrade, JX upgrade platform. Yeah. Because when we update a new version, we want to tell people to kind of update their client. Yeah. Yeah, we probably want that in JX state as well. We've got problems just to say the version's on our line. Because I think it's quite easy, because it's on time. Yeah. I was hoping HALM 3 would just be a thing when none of this matters anymore, but it's not really going to be a thing for a while. Yeah. We should fix it. Sorry, James. On you go. Is anybody really interested in what I've been doing? Yeah. There we are. Share the pain. So I did a little thing. One of the things we've struggled with in the past is ingress rules. So if you need to change the domain, for example, out of the box we create a cluster and we default to nip.io. Now, if you want to change to a real domain, it's kind of quite tricky to be able to start and do this. Quite a number of things you have to change. You have to delete your ingress rules. You then have to update the configuration for exposed control. You have to remove annotations from the service so that the external URL is updated. And there's also one of the other things we wanted to do for ages is enable cert manager so that you can get turned on HTTPS and TLS so you can get automated signed certificates generated for your ingress rules. It's pretty much well done. I've been testing, caught a couple of bugs, which I'm just adding some tests in. I've said it for the last three days, I'm that close. But I really feel it is. I'm not going to sleep tonight or there's going to be a little wine that will help me tonight and I'm sure that'll fix it. But then once that's done, then it is going to be fully focused on trying to get Proud in and tie up to work as Gareth's been doing around multi-cluster terraform. Now the World Cup is finished. I'm pretty confident we're going to get back people to sleep. What World Cup? No idea what you're talking about. Is that the football? I've just focused on the Euros, really. They're going to be awesome. Just two years to go. It's going to be brilliant. I'm going to start watching cricket. No. No, that's a bad idea. Cool. Any questions anybody has or any other updates people want to give? Okay.  Yeah. Should we call it a day? Yeah. Let's wrap it up. Thanks everyone. Thanks guys. See you soon.


We see the growth of organizations operating at the global scale or at the transnational international scale. Lot of factors have contributed to emergence of globalizing companies and globalizing companies also pose an important question to the field of OD about how to implement or plan the OD interventions at the worldwide scale. There are lot of opportunities for organizations to go global. For example, lot of international treaties are actually facilitating the emergence of the global marketplace. Because of the increasing competitive pressure and pressure to increase the revenue, increase the margin, to achieve the economy of a scale and scope, organizations have to look for market other than their domestic markets. Trade barriers are also coming down. We have seen lot of regional cooperation moments are emerging in different parts of world like European Union or ASEAN. These are the places where the trade barriers are coming down and lot of opportunity is coming up for the organizations to scale up their operations beyond their domestic market. Global advancement has fueled very significantly the process of internationalizing the business operations, IT technology, technical communication technologies and many others have contributed to expand the operations of businesses beyond the domestic boundary. But the process of globalization also has many challenges. The first challenge is choosing the appropriate strategic orientation. We will shortly look at the different strategic options available for organizations to reach out to the international market and those options have to be chosen carefully. Next challenge is managing their business operations across the culture. Human societies have different culture and accordingly organizations have to modify their various processes, processes related to supervision, human resource management, marketing, even some time operations. Though the global marketplace is emerging, the regulatory framework in different countries and different economies are still very different. For example, labor laws or policies related to human resource management, policies related to the environmental management or the functioning of the business itself might be different in different countries and in the process of globalizing our operations or in the process of extending the businesses beyond the domestic market have to be, have to take care of this difference in the regulatory framework. Last but not the least, controlling and coordinating the operations, marketing operations, HR operations and quality or production operations all these have to be managed very consciously in the process of scaling up the operations at the international level. Before we conclude our discussion on the appropriateness of the different OD interventions or the need for customizing the different OD interventions in cultural according to the cultural and economic context, we must recognize that the quadrants presented here are not the watertight compartments. That means an organization which is stereotyped here in any of this quadrant, we can have with the extra effort have OD interventions which are here explained as more suitable for another quadrant. For example, an organization operating in society having the moderate level of economic development where in the cultural fit with the OD practices also low according to this matrix is SAMCO operating in Brazil. So, if we go by this stereotype it is it seems to be difficult to implement a highly participative OD intervention in organization like SAMCO which will fall in this quadrant. But thanks to the leadership of the SAMCO in last few years, they were able to implement lot of participation based OD intervention. So, this kind of intervention according to this quadrant can be more conveniently be done in the organizations operating in this quadrant which is quadrant of high economic development and high cultural fit with OD practice. Similarly, China is classified and lot of Chinese company can be classified in this quadrant. That means the informal rituals will be having more importance the power difference is higher. So, the participative OD interventions or more team based OD interventions or more confrontation based OD interventions may be difficult to implement in this in the organizations operating in this quadrant. But we see lot of Chinese company have shown and have very successfully implemented change process which are theoretically considered more appropriate in this quadrant. So, what I mean to say is that organizational resolve and supportive leadership can help the organization to come out of this stereotype and even in the quadrants other than this one where both these elements are high organizations operating even in other quadrants with the help of a strong leadership and organizational resolve can implement more sophisticated OD interventions which are considered to be more suitable in high high quadrant organizations high high quadrant organizations in this matrix. Internationalizing the business operations depend on the need for integration across the different regions and need for responsiveness towards local market. It is reflected in this quadrant. This quadrant is drawn between two continuum. One continuum represents the need for local responsiveness and another continuum shows the need for global integrations of the business operations. They can be low or high and at the intersection of these two continuum we get four types of internationalizing the business operations. We need to understand how and what are the different modes of emergence of the multinational corporations or transnational corporations in order to understand what are the most appropriate OD interventions. So, first we will look at what are the characteristics of the internationalized operations of a business organization. They are not of one kind they can be at least of these four kinds. Let me quickly explain the these types and then we will look at how the different kind of OD interventions are appropriate in these organizations. So, our first example is of the international orientation. So, organization with the international orientation are more interested in extending their market without bringing about any change or much change according to the local market or the market to which they are extending their operations. Our example is a famous ice cream Hagendaz this is available in the metro cities in India and the product is not customized or the pricing also is not very significantly changed in comparison to what it is and it is a American market. So, in the international orientation generally organizations create marketing sales office in different companies sometime they implement some operations as well primary for the assembly or some basic work the core product generally comes from the home market or the home company and there is no customization is done according to the local requirement. In fact, organizations adopt the international orientation generally when they see that product is acceptable in the foreign market almost as it is. Another type of internationalizing operation we will see in the multinational orientation. Multinational orientation may operate in the same product category, but they customize that product according to the local requirement. We have given example of PNG the laundry product of PNG have to be customized differently in different environment. That means, laundry products will be different according to the hardness of water found in that market environmental regulation prevalent in that market the kind of washing machine being used or the uses of washing machine itself how widespread is the use of washing machine all these factor will determine what are the kind of products will be relevant or marketable in a particular market. So, they create a regional structure or divisions according to the region or the country. Region or a country has fairly independent operations operations in terms of production human resource management financial resource management etcetera. So, they have to coordinate with the parent organization, but they also have to customize very significantly to the local needs. So, another type of internationalized operations we will see in the form of global orientation. Here we look at the example of Nestle and Microsoft. What is special about these we will see the nature of product is almost same in across the industry, but their operations are very well integrated across the different markets. We will see the advertisement or positioning of the product or benefits of the product are same because the very core of the product is same, but the operations are handled in a very integrated manner with the global operations. Then we have examples of transnational organizations organizations having very high transnational orientation. Generally these organizations are designed according to the product categories. At one level their operations are very well globally integrated. At the same time it has a very high responsiveness towards the local market. The major characteristic of transnational organizations are that their operations are very well optimized their resource allocation are very well optimized according to the different markets they are operating. Generally these organizations are designed according to the product category and product category heads of markets operating in different region report to the global product category head and the resource allocation happens across and that across the globe and that resource allocation is can be in the form of financial, material, R and D and human resources as well. Now we will look at how the worldwide strategic orientation affect the kind of OD interventions implemented in these organizations. In the international worldwide strategic orientation they focus on the foreign revenue with the existing product through centralized international divisions. Most predominant OD interventions are strategic planning like search conferences and related activities. OD intervention at one level are focused on identifying the appropriate market and choosing a simple vehicle to reach out to that market. Generally the international operations are handled by the people coming from the head quarter and they are given cross cultural training. Not only the employees who are going to take the responsibility of international operation are given the cross cultural training their families are also given a cross cultural training. In the multinational worldwide strategic orientation organizations aim at selling the tailored product suitable to the local needs we gave the example of PNG and that is done through specialization through decentralized operations and centralized planning and geographical divisions. In these kind of organizations culturally aligned OD interventions or OD interventions aiming at building the cultural alignment of the division and the corporate office are very important. Intergroup relationship building between the regional groups regional management team and the global management team are also a very a very important aspect of OD intervention. Management team building formation of the strategic alliances, customized reward system these are some of the strategy and HR process related interventions generally you found in the organizations which have multinational worldwide strategic orientation. Organizations with the global worldwide strategic orientation and you might remember we gave the example of Nestle and Microsoft where the product are very similar and operations are very well integrated and that is a process of achieving the economy of a scale. In these kind of organizations we find the OD interventions which are related to building corporate vision because they want subsidiaries operating in different markets or different geographies must work as per the corporate vision. Intergroup relationship building is a is again a very important OD intervention here since the nature of product is same. They also aim at transferring people from one region to another region and in this kind of scenario extensive selection and rotation and the cultural development becomes very important for these organizations. Then comes the transnational organizations and you might remember we look at the example of G. Here the tailored products are provided through integration and decentralized worldwide coordination. So, a lot of business decision is taken at the local level and the synergy is being created at the global level that is one of the very important features of the transnational organizations. Transnational organizations generally operate with the global metrics or the network structure. Metrics structure you might remember we discussed in the organization design class wherein we looked wherein there is a local head to which functional heads report to and functional head also report to the global functional head at the corporate level. In the in terms of the OD intervention they are also building corporate vision is a very important OD intervention. In this setting as well the intergroup relationship is also important, but at the same time in transnational organizations recruitment happens for the large organization and not generally restricted to a particular region or division. So, you will see in organizations like G global leadership programs they recruit people not only to operate in a particular region or particular market, but they want to but they recruit at least the key talent which may be related to some product category, but they should be capable of operating in any part of the world. That is why you see transnational leadership development programs in a very beginning of the career is offered to the GE executives. As a result of that in the recruitment process not only local offices are involved not only the managers or the heads of the division of the of a particular geography or particular market are involved global team also play an important role in the decision making about the recruitment. So, in this session we looked at the cultural nuances of OD intervention, we looked at how the different cultures differ according to the shared norms, values, beliefs and we also had some discussion about how to customize the OD interventions according to the societal culture and the organizational culture. Next we looked at different flavors or different types of the worldwide operations and which are the appropriate OD interventions in international, multinational, global and transnational organizations. Thank you.


Good morning, my name is Commodore Phil Nash, I'm the naval attaché from the United Kingdom based here in Washington DC. This is my first time at the International Sea Power Symposium and my second time in Newport. It's been an amazing week, Newport's such a hospitable, beautiful place steeped in US naval history, it's been fantastic to share the week with so many other colleagues and Chiefs of Navy from around the UK so the message on behalf of the First Sea Lord from the UK is a big thank you for drawing everybody together against a pretty difficult background over the last 20 months. It's been absolutely fantastic to make reconnections with old friends and to meet many new friends, so important for so many senior leaders, thank you.


Okay, I'm going to move us along then. It's time for the reading of the conflict of interest statement. This applies to the applications that you will review in closed session tomorrow. We're required to do this in the open session. You must leave the meeting room when applications submitted by your own organization are being individually discussed. In the case of state higher education or other systems with multiple campuses that are geographically separated. Own organization is intended to mean the entire system, except where a determination has already been made that the components are separate organizations for the purpose of determining conflict of interest. You should avoid situations that could give rise to charges of conflict of interest, whether real or apparent. For example, you should not participate in the deliberations and actions on any application from or involving your spouse, child, recent student, recent teacher, professional collaborator with whom you've worked with closely. A close personal friend or scientist with whom you have had long standing scientific or personal differences. The GRI staff will determine the appropriate action based on recency, frequency and strength of such associations or interests, either positive or negative and will instruct you accordingly. In council actions in which you vote on a block of applications without discussing any individual one. If you vote on a block action, for example, your vote will not apply to any application from any institution fulfilling the criteria noted above. I think we will send you a conflict of interest form electronically and ask you to sign it and return it to comfort Brown. We have one more action to perform today in the open session, and that is, we have a retiring class of council members.  And in fact, they agreed to extend their service by another 180 days, because of some log jam issues we've had getting our council nomination slides approved by the secretary's office. So we're doubly indebted to this group for their dedication and service to NHGRI. We're fortunate that we're not together in person. But Eric, do you have the lovely parting gifts that we have for you are these glass obelisks, which are actually quite attractive. And they'll be in the mail along with a certificate. So it's a trust us thing will receive them eventually understand the postal service and then slowed down, but it's a treasure worth having. Our retiring members are Jeff. I for one don't want to envision a council meeting without Jeff. There's just a soothing authenticity and trustworthiness about Jeff that puts me at ease. Jeff spent on workshops, working groups, consensus panels that go back to the 1990s. And we're going to serve us on NHGRI review panels. Jeff always seems to be in the room and we're all better served by his presence and involvement. Thank you Jeff. I'm sure we'll see you again at another NHGRI event. Jonathan Haynes. Jonathan is the classic example of how council members get pressed into service to advise NHGRI on a broad spectrum of research areas. We recruited him to serve on the external advisory board of the large scale sequencing program. Then came data sharing databases and model organisms, functional genomics, integration of genomic data in clinical setting. And let's not forget the reason we recruited him to council, which is his knowledge of statistical genetics and association studies. Jonathan, you're always, you are an executive secretary's best friend. Trey Eideker. Trey, thank you for your wisdom on all things related to computational biology, modeling and data science, including your time on the data science working group of council. You've always been a trusted and reliable voice on the council and we will miss you. You've always been a trusted and reliable voice, but certainly not least, we come to Sharon Plon, the self-proclaimed Girl Scout of the council. Sharon, I don't remember in what context you brought that up, but you can always count on Sharon to give you the unvarnished truth and to keep us honest. She definitely tells it like it is. And then someone else is going to take charge and take on the role of starting council discussions because as Sharon's hand always goes up and there's always good wisdom that follows from that. So thank you for your advice on clinical genetics and all aspects of general medicine research and your contributions to NHGRI streaming program. This has been a great class. They're not only experts in your field, but they've been wonderful people to work with. And your devotion to NHGRI is duly noted. We are very, very grateful to you. So thank you. God bless. And I look forward to seeing all of you on a peer review panel very, very soon. Thank you. Any closing comments? Otherwise, I'm going to ask you to gavel us to conclusion. No, I'd like to thank everybody. We had a lot of concepts to get through. The discussion was excellent. All of you stayed energetic and helpful to the end. Much appreciated. We do have more work to do tomorrow, but I will certainly be happy to gavel the open session closed, Rudy, if that's what you want me to do. Yes, indeed. And we'll meet tomorrow beginning at 11am East Coast time. And again, there'll be another link that should be in your outlook calendar. So see you all tomorrow for closed session. Thank you very much. Have a nice evening. Bye.