Hello, everyone. Welcome to the session today. Today's topic, it is building multi-tenant routing and scaling with Envoy. I'm the speaker in Impen. I'm from the Amazon Web Service. Happy to be here to talk about the topic with you guys today. So a little bit self-introduction about myself. I'm the senior engineer from AWS based in Seattle. I'm the founding engineer of the AWS AppRounder. I participated in building the still small product from 0 to 1 and we did in the 2021. The product I'm working activities closely in the AWS is the including AppRounder, the Amazon ECS, Elastic Container Service, the AWS Fugate, which is focused on the serverless containers compute and Elastic Beanstalk. So my expertise I will focus on the cloud native containers and the serverless compute and open source. So I'm also pretty passionate in the open source community. I'm the founder and the maintainer of the cloud native and the serverless meetup community. You can find me in the GitHub and the Twitter. I'm pretty active there. So I want to show a little bit the motivation to have this talk today. I want to show the journey from the end user point of view by using NWI. I want to also give back to the community to show the lessons learned and experience by using NWI as a builder and hope it brings value to the community and help the product growth. And I also want to use this opportunity to thanks and appreciate all the support from the NWI community no matter it is internal or external. By the way, Amazon has an internal interest group for the NWI proxy with like around 160 people. I'm the founder and maintainer of that channel. I also want to send to the platform and I want to use this platform for the future discussion and communication. I'm a fan of KubeCon for years. I'm really happy and excited to be here to talk with you guys for the topic. A little bit of the introduction for AppRounder. As I mentioned before, so AWS AppRounder's GA in the 2021 is a fully managed service that makes it easy for builders to quickly deploy container web service and APIs at a scale with less and zero infrastructure experience required. Here is our official website. Feel free to check it out. So from the user point of view, the AppRounder provides a managed experience for customers app hosting, request the routing, load balancing and auto scaling, networking, the ingress, egress, both observability, CI CD deployment and the safe deployment and more. So customers just need to have their GitHub source code or the ECR image, private or public and deploy the image of source code connection to the AppRounder. AppRounder will directly make it up and running for your web app and or API service. And AppRounder going to output outcome and public URL for clients to access. AppRounder is a multi region supported and it is mainly focused on API and a web server. It accepts the HTTP request and it supports a concurrent request. And it also focus on the long time running servers which is today's world most popular and the typical traditional workloads. To build the AppRounder, to support the management user experience for AppRounder, including the request routing and load balancing, what's the responsibility of the request router? You may ask. So AppRounder needs a request router on the backend to help online multi-tenant request routing in different AWS regions. And it provides the URL for customer to access and online surrounding application instance for their containers. And then we need a request router to be in the middle between the URL and application instances to do the routing, to do load balancing and to do the traffic splitting. Product requirement. I like this because this is the motivation. Multi regions, region-expansable. Of course, we need to make sure we can build new regions all the time with the solution, with the existing architecture. You need to be multi-tenant. We are serving hundreds and millions customers as a platform and a solution. So we need the request router to be multi-tenant available. Auto scaling, we need to be scalable and we need to fail because in the future we're gonna face more and more traffic and workloads. Safe deployment. Of course, it is one of the basic requirements for the customer product. Observable. It needs to be manageable and provides necessary insights. So with the product requirements, the technical challenges will be obvious. They need to be high throughputs, high performance, high availability, high reliability. Think about your website. You don't want it to be down for a couple of minutes or seconds even. Extensibility. Of course, we need to build more and more features on top of it. Security. Large number of concurrence connections. As we mentioned, we accept the concurrent requests, which means for the request router, there may be a large number of connections happen in the same time. Observability. It's different with the product level of observability. We need the request router also provides observability and visibility for builders to understand what's going on in the middle and what's going on for the application activity and workloads. So we think about the on-way-based request router. On-way-based on open source edge and service proxy designed for native cloud applications. It provides features like auto process architecture. It is pretty lightweight and portable. HTTP layer 7 routing. GRPC ready and easy for builders to build the microservice around it. Basically in class observability. Provides necessary visibility for the system and service on the line. Microservice friendly. And it also provides managing the open source version available by the AWS app mesh. So we choose on-way as our request router base. I want to talk more a little bit the auto process architecture on-way provided. So on-way works with any programming language. Friendly, we can write the application in Go, in Java, in C++, and any other languages. On-way can bridge the gap between all of them and its behaviors identical. Regardless of the applications programming language or the operating system they are running on. Wiper owner, we are using protobuf and we write the service in C++, in Java, in golden. They can iterate with on-way with no gap and it break out all the boundaries of programming languages. The auto process architecture is beneficial as it gives the consistent see across the languages and application stacks. You can get an independent life cycle. Talking about building the on-way base request router. So first, the building blocks. So on-way supports the bootstrap configuration which is the key to make on-way up and running. Here we use the dynamic resources because the application instance is changed all the time. And looking at the architecture this is a very typical on-way usage architecture you can find in any way. You can, you should have a control plan to pass through all the configurations you need for on-way runtime. We will have a log and a metric system to collect and aggregate all the data and insights from the on-way runtime activity. You should have a UN streamer which is stream out all the outputs of the on-way activity and workloads. Talking about on-way itself. There are four building blocks for the on-way itself, internal. So first one is listener. Listener is the name network location. For example, the IP address plus the port. So on-way receives the request through the listeners. Roles. Roles is the configurations in the HTTP Connect Manager filter. Matching incoming requests like URI and handlers and define where the traffic is sent. Clusters. Cluster is a group of upstream hosts that accept the traffic for a route. List of hosts or IP addresses on which service is listening. Endpoints, which is a service application instance and the upstream IPs. We will talk about them one by one. First, the listeners. Listener is the component which is listening all the external HTTP requests and will operate on the page payloads. Received request and wrote to the next component block. Roles. So on-way will be the CRUX of load balancing and layer seven request proxying layer. It maintain a map of the service URL to the application instance IPs and wrote incoming requests to proper endpoints. It also mapping a domain from the service URL for example, halloween here to the upstream endpoints. Clusters. You can treat on-way clusters as the application service. For here, each service version going to be assigned an on-way cluster and the configurations will be updated through the on-way cluster discovery service. The short name is CDS. Endpoint. The upstream endpoints updated will the on-way endpoint discovery service. The short name is EDS and it supports the dynamic and auto-scaling and also covers health check which is part of it is the on-house endpoints reaper. I want to talk about the middle layer between the listen and the loads. It called HTTP connection manager. Show name is HCM. HCM, it is a network layer filter. It translates real bytes into HTTP. It handles access logging, request ID generation, tracing, handler manipulation, retry policy, timeout setting, traffic weights, and road waiters. For HCM, the road used the information from the incoming requests. For example, the host or authority handers and matches it to an upstream cluster through the virtual host or routing rules. Activity filters use the road configuration that contains all the road table for the HTTP routing. An set of handers can be specified to match the request. The router checks the request handers against all the specified handers in the road config. Some match includes the range, present, string, inward match. HCM filters also support the query parameters matching, the TRS context matching, and the GRPC routing matching for routing. And it always supports traffic splitting to different roles within the same virtual host. We can split the traffic between the two or more, upstream clusters. The traffic split happens on runtime, percentage, or weighted clusters. So, ANWI supports two types of the health check, active and passive. With active health check, ANWI periodically sends a request to the endpoint to check its status. With the passive health check, ANWI monitors how the endpoint responds to connections. You can use either of them depending on your case. Load balancing. Load balancing is very important because it is a way to distribute traffic between the multiple endpoints in a single service and a single upstream cluster. The reason to distribute traffic across multiple endpoints is because we want to make the best use of the resource. It helps the cost-effective for the product oil for your applications. Retrides. ANWI allow the retries and allow the setting the retry policy at the virtual host level at the HTTP Connect Manager and also the road level based on the conditions. For example, if there's a 429 status codes return from the endpoint, then we can set the retries. Then retries will happen stiminously on the upstream of ANWI. We can set the retries on the status code. We can set the retries on the priority level depending on your use case. Circuit breaker. This pattern prevents additional failures managing access to failing services. It allows us to fail quickly and apply back pressure downstream as soon as possible. It is best practice. We can configure breaker threshold for each road priority separately and globally. Time mode. ANWI supports multiple configure robot time modes depending on your scenarios. For example, there's a request time mode specified some amount of time ANWI waits for the entire request to be received. There are also idle time modes represent when a downstream or upstream connection gets terminated if there are no active streams. Rate limiter. ANWI supports the global and local rate limit to the upstream endpoints. We finished ANWI itself's building blocks. Let's step back and see the old architecture, the control plan component. Generally, you can view the customized control plan for ANWI runtime. It will responsible for service cluster discovery, CDS, and service endpoint discovery, EDS. And it was also responsible to passing all the route or other configurations necessary for ANWI runtime activity to the ANWI. It can be built as the GRB server listening on a defined port. ANWI needs to be updated with latest service information at runtime. So ANWI discover dynamic resources by curing the management servers using XDS which is full name is the discovery service APIs. The dynamic resource I'm in here including the clusters and endpoints. ANWI request resources with subscriptions by initially GRPC stream from ANWI client to the management server. Talking a little bit more about cluster and endpoint discovery, cluster going to be configured on ANWI through the CDS discovery service API and endpoint going to be updated through the EDS discovery service API. All of these APIs are open source. An app runner use the CDS API dynamically update clusters and the EDS to update endpoints per cluster. Location metrics is the other component which is important to collection all the insights and activity data. ANWI can configure exposed metrics locally and the static sync that received the metrics immediately from ANWI and do all the collection aggregation. For the metrics such as 2XX, 5XX request account can even truly forward it to logs and metrics service. This can be built as a GRPC service listening on defined port and all the collected metrics and logs can be routed to the popular cloud native or Adobe solutions. For example, the cloud watch which is for metrics and logging and then managing the service for permissions for monitoring and open telemetry or graphona or x-ray. The last component is the event streamer. It is responsible for reporting service status and health check results for the ANWI runtime. It also can be built as the GRPC service listening on defined port and all these health check and status results will be used in the supporting service like a safety deployment, blue-green deployment or automatic scaling, load balancing and request routing. So we finish all the functionalities of ANWI related ecosystem. Let's talk about how to make ANWI based system in production readiness. So besides the functional readiness, we need to make sure the capacity management is under control. It includes the CPU and the memory consumption because it is high performance, high throughput. You want to make sure there's nothing which is out of the expectation. We also need to make sure the performance and the scalability for the ANWI system. And at the same time the security, for example, DDOS, how you prevent the death DDOS which is in the place. And also, ANWI operational readiness items including the monitors and lamps. To ramp up, so the ANWI brings values in building a very solid request routing with the many features. High throughput and performance requests, routing and load balancing, portability, extensibility, reliability and availability. Of course, it provides very, very good observability. Easy for builders managing the online system and extends the system to build new things on top of it. With all the ANWI related topic finished, I also want to introduce a little bit on the Apronur's public roadmap. Apronur as a product has a public roadmap in the GitHub. We receive and collect customer requests and also we interact with customer directly, actively on the roadmap to show our plan, show what we are working on. There are also a couple of items which is related to request routing and poxing in the Apronur. For example, the private service support, additional ex forwarded hander support and WAF support and configuration timeout support. We are happy to receive your feedback and feel free to check it out and let us know how we can do better. This is all the things I want to talk about here. So thank you again to be in the session and also I will leave the rest of the time for the Q&A. Thanks.


Okay, very good morning to you. It is Tuesday 21st of December. Hope you're doing well and gonna delve straight into The news that matters for the day ahead and even though there's quite a quiet calendar for this week in the run into Christmas There's still quite a lot of news and subsequent market movement happening at the moment It's all really based around the continued emergence of Omicron on a global level and the potential that that can have for further restrictions in time and So when we look at the close on Wall Street yesterday the S&P was down about 1.1 down One and a quarter percent the Nasdaq down a similar margin and actually for the S&P 500 It's the biggest three-day drop that we've seen since September However, one thing is looking at the charts this morning. We have had a bit of a reversal Really commenced despite that weak handover from the close on Wall Street Equity index futures going to the European open trading quite firmly higher this morning Oil's also seen a decent bounce since finding a bit of a flaw from around 5 p.m. Last night So the ship has steadied after it got rocked a little bit by really two major things not just Omicron But the other thing of course that came out yesterday well over the weekend is continued resistance from the Virginia senator Joe Manchin and as you can see here, this is a picture of him and Yesterday Joe Manchin's apparent no vote on the bill back better bill Goldman Sachs downgraded its GDP forecast for the first quarter from 3 to 2% Second quarter trimmed from three and a half to three percent and third quarter from three to two and three quarters percent They said Manchin cited the country's existing debt levels the reemergence of COVID-19 Rising prices for consumer goods. These were all his rationale for rejecting that bill So at the moment what we're seeing is economists are having to re-evaluate then the kind of medium term Economic growth picture on the back of how impactful Omicron is likely to be and I'm impeding economic activity Layering in an inability for this substantial bill to pass or at least be delayed in the US And that's what led to the majority of the weaker sentiment yesterday Couple of other things that are very interesting to update on the global Omicron situation This was an interesting report I did share on my Twitter account if you want to read the full kind of piece But this was looking at researchers at Imperial College London who basically compared just over 11,000 people With confirmed or likely Omicron infections with nearly 200,000 people Infected with other variants and so far according to a report issued ahead of a peer review and updated yesterday They see no evidence of Omicron having lower Severity than Delta by either the proportion of people testing positive who report symptoms or by the proportion of cases seeking hospital care after infection and of course this would be Kind of leaning on worse cases scenario given the highly transmissible Qualities that this latest variant is showing if it is indeed then appears no less severe than Delta That's going to be highly problematic going forward given the likelihood as many officials are saying that case rates Probably in the UK or even in the hundreds of thousands at this point in time irrespective of case rates numbers being Factually reported around the 90k mark of the last few days Other things then are what's happening in the US and As you can see here the Omicron variant accounted for 73% of all sequenced Covid-19 cases in the US surging from around 3% last week. It's gone from 3 to 73% in a week These are according to latest federal estimates One thing I'd say on that point is as dramatic as that headline sounds There was I would say somewhat inevitability that this would happen I would say the fact that it's gone from 3 to 73% is pretty rapid by any stretch and and so Was adding fuel to the fine some of the yesterday's move and deterioration and sentiment that we were seeing one thing to bear in mind though is the Moderna did say yesterday that a third dose of its vaccine increased antibody levels against the variant and Pfizer-BioNTech have said lab studies show a third dose of the vaccine also helps neutralize Omicron and what we're looking at here on this table is Then the composition of the vaccine rollout that we've seen in the US So very different to here in the UK, which is heavily Built around the AstraZeneca drug in America It's pretty much exclusively Pfizer and Moderna Moderna very less adopted elsewhere on a global level You can see here proportionately From a percentages roughly as a reference you're looking at around 38% of all vaccines deployed in America on Moderna's ones And some of those antibody level results from what a booster gives you in protection were quite encouraging yesterday and Pfizer makes up around just shy of 60% so actually in actuality This is quite positive things, but it's all dependent on then really having a successful booster rollout strategy In order to give people the relevant protection against if it is proved to be as severe as Delta the latest Omicron variant So overall I'd say just on this top-level situation when you look at the charts this morning Yeah, as I said things have started to recover I think that yesterday obviously the the negative kind of selling pressure in lights of global indices as well as oil prices kind of came in as the market opened and I think it's kind of just Reaclimatizing to the Realization of how impactful I think Omicron is going to be but I think that that's kind of faded a little bit since that point in time I think it was probably just somewhat Exacerbated by kind of going into the run-in into Christmas perhaps a touch slightly lower Volumes and so forth, but overall at the moment This is a situation to be monitored obviously quite closely But the ship has steadied somewhat but it'd be interesting to see how we go throughout this session to see Really where the the lay of the land in terms of market sentiment is the other thing of course is in the UK And so a quick update Boris Johnson, of course yesterday You probably would have read by now held off introducing stricter UK coronavirus rules to clamp down on the new surge happening here in The UK, but he did leave open the prospect that they will be needed soon And obviously he's been under increasing pressure. I just mentioned the Imperial College They've typically always been quite bearish in terms of their assessment of how impactful The coronavirus is going to be Sage though themselves have also been quite insistent on taking more strict protocols in order to contain the latest outbreak Problem is of course politically Boris Johnson under multiple pressure points at the moment cabinet officials highly Sensitive to the idea of further restrictions and this has been shared by cabinet ministers, which are quite a large number were Satisfied let's say with the actions or muting of talking about doing further protocols going forward To give you a bit of an idea of other factors as well in the UK the number of Londoners hospitalized with coronavirus is Rising sharply at this point in time putting more pressure on the government, of course And there were more than 1,800 people in hospital with the disease on Monday an increase of a third in just a week According to NHS England This came yesterday in the context of a report leaked by the Daily Mirror Which talked about the government may introduce new restrictions from December 27th So just after Christmas the restrictions being considered our Step two of the country's earlier coronavirus response Which would include pubs restaurants only being able to serve customers outdoors a Ban on indoor mixing between households and a maximum of six people or two households being allowed to interact outdoors My my belief is that that will happen and that will happen On the 27th. I actually think that that's given the way that the coronavirus Situation is developing at the moment particularly in the UK. I think that is very much like to happen politically He's not really now got the timetable To be able to make any necessary changes or the political appetite within his own party to do anything now but Probably these numbers and the situation and the pressure certainly on the hospitalizations and NHS in the in England It's going to get materially worse between now and till the 27th Requiring then I would say that that level of restrictions probably is going to come in at this point in time The cabinet they are going to meet again this week In fact further dates on the severity of the new variant is expected on Wednesday So still on the lookout for any further updates as we go through but I'd be very surprised if anything more Onerous comes in prior to Christmas as I said most likely on the 27th The other thing final story then just focusing on the big things that are really moving at the moment is the Turkish lira You obviously would have seen this for quite a while Persistent weakness in the currency on over the somewhat unorthodox approach of monetary policy that's being deployed by President Erdogan well the lira at one point fell as much as 11 percent from Friday's close to touch 18 36 per dollar its weakest level ever. I think when Eddie and I did a Chat and a video we put out on YouTube a few few weeks ago It was trading around 12 and that was causing a big big shock at the time So it's now in excess at 18 or was yesterday and that of course is the weakest level ever against the US dollar And that followed weekend comments from the president indicating he wants to see rates go even lower And this comes after he's been continuously cutting rates at kind of hundred basis point clips irrespective inflation Surging at this present point in time But new measures announced by the government yesterday helped put a floor under the currency which proceeded to rally close to 47 percent from its intraday low to touch 12 28 per Greenback which you can see here. So this is the intervention that came in This was when we were below or over 18 And then because we're looking at lira's per dollar here and then the aggressive move up that we've seen And we're tracking more back towards the 11 mark at this point in time So the currency actually ended the day yesterday about 24 percent stronger than Fridays closed and did mark the biggest one-day gain since 1983 The latest major shift, I guess if you're asking what was that? Came after government now steps including the introduction of a new program to protect protect savings from Fluctuations in the local currency the government will make up for losses incurred by holders of lira deposits should the lira's declines Against hard currencies Exceed interest rates promised by the banks Erdogan said cherry a cabinet meeting in Ankara yesterday the Turkish Bank Association head. Let's give you a bit of Kind of context said around one billion US dollars were sold in markets after Erdogan announced those new measures So for the moment Yeah Intervention has taken place probably somewhat inevitable not uncommon when you see it in EM markets to see this type of currency depreciation however Intervention comes at a severe cost and it's just going to be interested to see then the Subsequent direction of this price over the coming weeks Obviously if it continues to then re-weaken and test these lower bound levels again Then how much firepower and appetite does really turkey have for the moment Erdogan's being stubborn He'll probably remain that way Until the very last moment and so those measures working for now But I guess we've got to wait until really into the new year to see the dust settle and where the land lies at that point in time Looking at Canada for today Very quiet as is per the whole week There's nothing really on the docket that really is going to raise your pulse rate at this point We've got European consumer confidence at 3 p.m. This afternoon API regular weekly infantry's later on aftermarket and 20 billion dollars in a 20-year bond auction from the US at 6 p.m. London time But that's pretty much it so with that I'll leave it there wish you guys a good day ahead See you same time tomorrow. Take care


Americans always have cared about the less fortunate and I'm sure it'll deeply gladden the hearts of many of you to know the kind of progress we've made during the past six and a half years in helping the poor. We have between four and six million fewer low-income families on the federal income tax roles. We've tamed inflation rates that were devastating the purchasing power of those least able to afford the basic necessities of life. That's right, we're taping it isn't alive is it? My fellow Americans, Americans always have cared about the less fortunate and I'm sure it'll deeply gladden the hearts of many of you to know the kind of progress we've made during the past six and a half years in helping the poor. We have between four and six million fewer low income families on the federal income tax roles. We've tamed inflation rates that were devastating the purchasing power of those least able to afford the basic necessities of life and reversed an upward spiral in the number of poor people that began in 1979. The official statistics released on Thursday show that the poverty rate is down for the third year in a row. The 1.6 percentage drop in poverty over the last three years is the largest sustained improvement since 1970 and median family income adjusted for inflation rose by 4.2% in 1986, the largest increase since 1972. All of us can be pleased with this progress. Pleased but not satisfied.  It's now common knowledge that our welfare system has itself become a poverty trap, a creator and reinforcer of dependency. And that's why last year in my state of the union message I called for an overhaul of our welfare system. Since that time, I've sent to Congress a carefully designed package of proposals that rejects the old federal approach of sweeping solutions dictated from Washington. The central point of our new proposal is outlined in our earlier study up from dependency and now embodied in our legislative proposal, the low income opportunity improvement act is a provision that will allow states and localities to test new ideas for reducing welfare dependency through experimental changes through carefully tested and evaluated demonstrations. This new approach can determine what does work in reducing welfare dependency. The National Governors Association met last weekend in Michigan. They gave substantial support to our plans to give them greater flexibility and they promised to work closely with us and the Congress. But while we must let loose the creative energies of our states and localities, I think there are some critical improvements we can make at the federal level. Under the laws now in place, all mothers who have children under age six are exempt from participating in work activities that several demonstration projects have shown can help aid for dependent children, AFDC recipients become more self-reliant. Fewer than one-fifth of all recipients now participate in work activities. We must lift this counterproductive exemption and thereby get early help to these women and their children before they become chronically dependent on welfare. We must also reform work requirements so that long gaps in school or in other work related experiences no longer occur. And so too, work opportunities for AFDC recipients must be expanded. We must give teenagers on AFDC who have not completed high school the opportunity to continue their schooling and older recipients to participate in employment and training activities. Two proposals we've sent to the Congress, grow or greater opportunities through work in AFDC and the AFDC Youth Training Initiative will allow us to do all of these things. So too, changes in our child support enforcement system can reduce welfare dependency. Parents who bring children into the world have a responsibility for these children whether they live with them or not. The administration is taking steps to ensure that states are able to do a better job in locating absent parents, establishing paternity, and collecting child support on behalf of AFDC recipients. We also have asked the Congress for new laws that would increase child support award amounts for both welfare and non-welfare families. Now the question I ask about any welfare reform proposal is, will it help people become self-sufficient and lead a full life or will it keep them down in a state of dependency? I'm afraid that several members of Congress have suggested some proposals that while claiming to require work related activities would make staying on welfare more attractive. Their misguided compassion would only bring more people into the welfare system, encourage them to stay on the welfare roles longer, and discourage work. For example, the Democratic House bill contains no demonstration authority at all, and another Senate bill, only a very limited one. AFDC work program reforms that emphasize early intervention to prevent welfare dependency, child support enforcement, improvements to provide children the help that they need, and demonstration projects that give us the information necessary to make changes in the national welfare system. That is my welfare reform strategy. I hope it has your support. Until next week, thanks for listening. God bless you.