All right, let's get started. So today, we're going to talk about assembly theory. So I will first give you an overview of what assembly looks like and what we are supposed to do in assembly theory. Then we'll try to study how to design the design doc, to write up the design document by learning how DamVM works and how by fully understanding DamVM. And then I will give you some key points when you write your design document, what to address, and what to explain in the document. So this is a summary. So it has coloring questions as you. And it has design document of which these two are due this Friday afternoon. So basically, you can split assembly theory into three stages. So first stage, you do physical memory management. You allocate free physical pages inside the kernel. And that's the first stage. And second stage, you design the user address space and also the page table. And you do with all kinds of TRB management and all that. And finally, you will do swapping. So in the very first stage, where you don't need to worry about user address space or page table at all, just allocate and free physical pages. In the second stage, you can assume you have enough memory. So when you want to allocate a physical page, use the function you implemented in the first stage. You can assume that you can always get a physical page. You have enough memory. And finally, you can shrink the memory size. Now you cannot assume that every time you want a page, you will get a page. There are circumstances where when you want a physical page, there is no free physical pages available. So you need to swap the physical page out to the disk. And also maybe later on, when some user program access that page, you need to swap that page in from disk to memory. So here, this is other three stages of assembly theory. You can do this incrementally. And all the functions you implemented in the previous stages that you will use in the last stage. For example, in the first stage, you will design many two functions. Allocate a physical page and a free physical page. So it's basically the two functions you write in the first stage. And then in the second stage, you basically need to design the user address space. Then you have your page table design. In the second stage, one main function you want to write is the VM fault. How to handle the virtual memory page fault or TRB fault. And finally, when you do swapping, there are two main functions that you want to write, which is swap out a page or swapping a page. So these are the three stages of assembly theory. And you want to do this strictly in this order. You don't want to do swapping first, and then physical memory management. Because you cannot do swapping or address space without a physical memory management block ready. So you want to do this in this order. First deal with the physical memory, then use address space, then swapping. So as you know that in assembly theory, you are supposed to implement the virtual memory system. So first of all, what is a virtual memory system? What's the interface of it? What's the functionality of it? So this is what you would expect from a virtual memory system, from either from a user program's point of view or from other part of the kernel's point of view. So our virtual memory system is supposed to manage all the physical pages. You can imagine, in this scenario, the resource is that the kernel managing is all the physical pages, physical memories. And in particular, this virtual memory system should be able to allocate a physical page and reclaim a physical page. It should provide such functionality. And also, the virtual memory system are supposed to do the translation between the virtual address space and physical address space. As you already learned from the lectures, every address you use in user space is virtual address. The user always uses a virtual address. And when you actually want to get the content from that virtual address, you need to translate that virtual address into a physical address. That translation is done or is managed by the virtual memory system. And in particular, you will need to manage the TRB for the translation. So this is the interface or the functionality of the virtual memory system. These are the functionalities that you should implement in this assignment. And let's look at the interface of each functionality one by one. So first of all, we have physical page management. For this, we have many two functions. That's important. Other or others are just some auxiliary functions that you write to help you do the job. So first of all, we have a function called vmplusdrap. This function is an initialization module for this virtual memory system. This function will be caught by the kernel in the bootup process to tell the VM, hey, I'm booting up. Do you have any death structures you want to initialize? For example, you may have a big static array. This is where you want to initialize that whatever death structure you use for the virtual memory system. And then we have these two very important functions called allocated key pages and free key pages. As the name suggests, allocated key pages are supposed to give you N number of pages where N could be one, two, or three, four. MSO, right? So this is the two most important functions you need to write for this, in the first stage, the physical memory management. So we also have a function called free key pages where the caller of the free key pages tell the virtual memory system that I'm done with the physical page. Now it's okay for you to reclaim it. You can wipe the content. You can mark the status of the page whatever you want to do. So these are the two interfaces that allow the virtual memory system to provide the physical page management to other part of the system, right? Then we have two TLB related functions called tlb should done or tlb should done all. At the first stage, when you do the physical page management, you don't have to worry these functions too much or for now you can just leave it empty. You can just do nothing in these functions. So but later on when you do user address space and when you have context switches, you may want to modify these two functions to do the job that the name suggests where tlb should done will close one tlb entry or flush out one tlb entry. tlb should done all will flush all the tlb entries but for now just leave it alone. And finally we have one function called vm thought. Again, in the first stage where you do the physical page management, you can just ignore this function. This function is important in the second stage where you want to manage the user address space. So far, when you're working on the physical page management, you don't have any user program running. So nobody will trigger a vm thought. So you can just leave this function alone but you will worry about this function in the second stage. Let's take a look at what the, any questions so far about this page, about physical page management? So as I said in the slides, the functions are defined in vm.edge. So this function basically defines the interface of the virtual memory system. So we have vmbluestrap, which is to initialize the destructors of your virtual memory system. Sorry about that. You have vmthought, for now just leave it alone when you're working on the physical pages. Then we have allocatedK pages and freeK pages. You can see that the allocatedK page is, where is that? Okay. Oh, sorry. I just keep this configured. So allocatedK page is take one argument which is how many number, how many pages that the caller want to allocate. And freeK page is take one argument which is where the initial address of the physical page. From this to what can you tell from the, what can you say about these two functions? So when I say I want to allocate four pages, can you just return me any number of arbitrary four pages, scatter around all the physical address space? That has to be continuous, right? So because allocatedK pages just return the start address of the first page. So implicit assumption of the allocated pages is that these pages has to be physically continuous with each other, right? Say I want to allocate four pages, then you have to return four continuous pages instead of just random four package, random four pages all around the physical address space. So similarly, when user want to free these pages, all you have or all you can get from the function is the start address of the chunk of pages, right? So this give you a hint, right? When you receive a freeK pages call, the only thing you get is a start address of the physical pages. A natural question to ask is, how do you know how many pages to free, right? That's something you need to consider when you do this physical page management. That's just one of the questions you need to consider. And finally, we have one more offline here. And finally, we have this TRB related functions, which I suggest you to leave it alone in the first stage. So this is the first part which related to physical page management. And the second part of the interface is related to user address space management, which basically the AS on scope function family, right? Some of them you become very familiar with in assignment two, like AS create, AS destroy, AS copy, which you use in fork, AS activate, which you use in child of four country, AS define stack, which you use in C's exact V, right? There are some other functions that you need to understand what they are supposed to do and why you may want to change the implementation of it. So this, so create, destroy, copy, activate, define stack is quite obvious. You know what it does.  The loadf are supposed to load that binary into memory, right? You do that in exact V when you want to execute a new user program. So AS define region basically tells the virtual memory system, right, that this user program has such a region with a starter address and with the length of the region. So the AS define region is the interface of the virtual memory system to receive how many regions that the user program has. And also prepare load is to tell the virtual memory system that the loadf function now are ready to load all this content into memory and complete load tells the virtual memory system that loadf is done with the loading part. So let's take a look at the loadf function and see how these functions are called. I don't know why I have it here. So let's open the loadf function which is in assembly you will find yourself that you need to understand the implementation of loadf to be able to implement the three functions correctly, which is define region, prepare load and complete load. So here loadf basically do load the user binary in a loop where it read all those regions. So loadf will first tell the virtual memory system that user program has this, this, this and regions. There could be a variable number of regions that user program typically a program has too, which is code segment and data segment, but some user program may have more regions or segments. So loadf first call AS define region to tell the virtual memory system where the region is. You have a virtual rest start, you have a memory size which are size of the region. You have the permission of the region. Different regions have different permissions. For example, the code regions are read only, right? You cannot write to a code region. They will overwrite the user's code. And the data region is read or readable and writeable, right? So later on when, so for example, when you get a page fault, say a user program want to write a virtual address. The first thing you want to figure out is that which region that virtual address belongs to, right? So that you can check the permission. For example, if you have a right page fault to a page, and then you figure out that this page is read only, then you should reject such page fault by, I don't know, maybe panic or kill the user program, right? So this is the info where you want to record or you want to record what's the permission of the region or what's every information you need for each region. So later on, when you have a page fault, you can check against that. Or another case would be you have a page fault on a virtual address, but you find out that that virtual address doesn't belong to any regions. Then what do you can infer? That's the invalid user address, right? You should reject that instead of allocated pages for that. So that's AS defined region, which basically tell you which regions are there. Then you have this prepare load, which just to tell you the virtual memory system is, just to tell you the loadf program is ready to load this content into memory. And why doesn't it matter? So there are basically two approaches to allocate pages. Say a user code segment is five pages. There are two options when you want to implement the virtual memory system. Whether you allocate these five pages on-demand, which means that you don't allocate any physical pages at all when users start running. Then whenever you have a page fault, you figure out whether or not it's a valid address, then if it's valid, you allocate pages at this time. That's a one approach which is called on-demand page allocation, right? Or another design option could be you just allocate, pre-allocate all the pages for the user, whether you don't care if the user actually access it or not, right? This is where, so if you decide to implement the second strategy where you want to pre-allocate all the pages for the user without triggering any VM fault later on, you can do it here. So this is where basically the loadf tells you that I'm going to load all the content into memory. I'm going to write all those pages. Whether or not I want to allocate the pages at this point or later on when there is a page fault, it's up to you. But at least the loadf system tells you that he's about to write to their pages. And then we have a lot, we read the content from the disk and load the content to memory, load a segment. And finally when the loadf is done, loading the contents or the segments because it's complete load, right? So this is how the other part of the system in this case loadf, we use the interface of the virtual memory. You order to implement those interfaces correctly, you need to understand how those interfaces are going to be used. This is a great example of how this interface will be used in practice. So we have the user address page related to virtual memory interfaces. Any questions about this part? This is, so the today's content is also the stuff you want to discuss in the design documents, right? So you know what to do, implement the virtual memory system. You know what's required, that's what we're doing. We want to identify what interfaces we want to provide, what the functions you want to implement. Then you need to discuss how do you want to do it, how to design all those functions. And today we'll learn that by seeing how DOM VM does that and how you can improve upon that. This is our way to approach the design documents. First identify what's there to be done and then see what's already there. So what's the DOM VM has already done. Then you can discuss what you can improve upon DOM VM. So this is the second part, which is address space related interfaces. So finally we have one more thing to implement for a simultaneously, which is called S-Break. S-Break is basically the syscall that user can use to request memories. This is related to the heap allocation, which you will do in the second stage. So you can read the menu pages of S-Break and figure out what S-Break is supposed to do and implement that. I assume at this point you are very familiar with all the syscall stuff. So you can play with it. So now we have finished going over all the interfaces or all the functions you are supposed to implement in a summary. Now let's take a look at how DOM VM accomplish these functionalities. DOM VM is also a virtual memory system. You need to do all we have discussed. So let's see how DOM VM does it. So in terms of physical memory allocation, DOM VM implemented these two functions. One is allocator capages and free capages. So the way DOM VM does it is really done in the sense that, suppose this is the whole physical memory. Suppose you have 16 megabytes of memory. This will be physical address zero. This will be physical address 16 megabytes. And when you boot the sys161, it will load the kernel in somewhere in the physical memory. The kernel will occupy some physical memories. Let's suppose this first p address is where, right after the kernel's pages. So this is basically the first available physical addresses. And then we have the last physical address, which is basically the size of the memory. So from the first physical address and to the last physical address is where the available physical pages or physical memories are. This is if you want to manage all the physical pages, this is the pages you want to manage. And how DOM VM manages these physical pages? Any idea? When I call allocator capages, say I want to allocate five pages, what DOM VM does? So what DOM VM do to give me those five pages? Yeah, so basically you have this available number of pages. Whenever I want to allocate five pages, it just forward this first physical address by that many. Say I want to allocate four physical pages, it just forward this to four physical pages and return this chunk to me. Next time, I want to allocate eight physical pages. Take forward this first physical address again here. Keep forwarding. When I call free capages, what does DOM VM do? Nothing. So I keep forwarding this pointer until this first physical address is equals to or larger than last physical address. At that point, DOM VM will just panic. So as you can see that, DOM VM really don't have the capability of managing it in a sense of allocate and reclaim. DOM VM can only allocate it for you. But DOM VM doesn't have a way to reclaim all those physical pages. When you call free physical pages, free capages in the case, DOM VM just does nothing. So this is clearly not desirable. We want to manage all those physical pages instead of just panic when we run out. We should be able to claim them. And actually, you can see that from the code of DOM VM. So take a look at allocate capages. What does it do? Call this magical function called getp pages, which stands for get physical pages. And you jump to getp pages. Well, DOM VM is certainly clever enough to know to use a lock to protect the access to the memory. Because this is one thing you also need to consider when you implement your own virtual memory system. Because this physical memory is system-wide, so you may have multiple threads that are called allocate capages at the same time. You want some synchronization primitives to protect access. In this case, DOM VM decides to use a spin lock. So it first acquires a spin lock. Then it causes function, which is called RAM-steal-memory. It just keeps stealing memory from the RAM system until it runs out. If you go to the RAM-steal-memory, you will see that's exactly what we have described in the slide. So it just forward this first physical address by that number of bytes and keep doing that until it runs out of memory. So if, here, in this case, the first physical address plus the size is larger than last physical address, it just returns 0, which means there is no more memory. In that case, allocate capages will return 0, and it will return 0 all the way up. And eventually, somewhere it will panic. So this is how DOM VM manages the physical memory. Clearly, the limitation is there is no reclaim or recycling happening here. And it just keeps stealing the RAM until it runs out. How do you want to improve this? How would you manage the physical pages? Clearly, you need to maintain some information about the status of each physical page. So you know that when I want to allocate, I just scan my data structure, figure out which pages are available, and return that. When somebody calls free capages, I want to mark those pages as available again. So next time, I can return them to some other program. So there is one data structure called call map, which is very important in physical memory management. Basically, call map is the data structure that has the information of the status of all the physical pages. You can just imagine it as kind of similar to the file tables or process tables. But it's different from that in the sense that the size of call map is fixed. Say you have 200 physical pages in total. How many entries are there in your call map? 200, fixed. There is no allocation and stuff. You just figure out how many physical pages are there and initialize the call map of their size. So you need to think about what information you want to keep for each physical page. For example, most basic information is whether this page is available or not. That's one flag to indicate the availability of the physical page. What about others? What other information you want to keep for each physical page? That we will discuss in detail next time. So by this time, I just want to give you an overview of 7.3 so you know what to write in the design document. That's certainly something you want to discuss in the design document, what the call map looks like. And secondly, how does the MVM handle user address space? So this is what the user address, virtual address space, looks like. We have virtual address 0. We have virtual address 0x and a median, which is the user space top. So inside that, user have a code region, which is a couple of pages, not data region, maybe one or two pages, and also a heap, which is variable and can group downwards all the way up. And at the top of the user address space, we have a stack that starts from the user space top and then group downwards. This is what user address space looks like. And let's take a look at how the MVM handle this stuff. So first of all, you can see what assumptions that MVM makes of the user address space. What is that? You can see it's V-Base 1, P-Base 1, m-pages, which corresponding to the first segment. V-Base 2, P-Base 2, m-pages 2, which corresponding to the second segment. All in two segments. So MVM assume that there will be only two segments in the user address space. If the user address has more than two segments, that's all right, MVM cannot handle it. It will panic or doing some sort of silly, right? So this is one assumption that MVM assumes that user program will always have two, at most two regions. And finally, we have a physical address called AS-Stack-P-Base, which stands for Stack-Physical-Base-Address. What does that mean? So this is a physical address, one value, which point to the physical base of the stack, right? So another assumption that MVM makes is that the size of the stack is fixed. And you will see that in OS16, when MVM I think is 12 or something like that, maybe not 12, but it's a fixed number. So MVM assumes that the user stack is fixed-sized, so that they can use one physical address to store the base of the stack. Say for example, this P-Base is OX at a medium minus four times the page size. In that case, MVM assumes that the user stack is at most four pages. If the user stack has some like infinite recursive cause, like which expanded the stack beyond that four pages, MVM will just panic. Because MVM assumed that the stack size is, the maximum stack size is fixed. And you can also see that it is caused some waste, right? Because you have one fixed value. Some user program is very lightweight. It will only use at most one pages, but you allocate the four pages for it. Some user program may have a heavy recursive kind of cause where it will require a larger stack, but the MVM cannot handle it. So MVM use one physical page size, one stack size for all user programs, right? That's not efficient. That will waste the physical pages. So clearly we want to improve it. So here you can see the limitations of MVM. As we have mentioned, fixed number of regions, too. Oh, another thing I want to mention is, do you see anything related to heap in the structure? No. Because typically a user program have at least a code segment, a data segment. Some user program may have multiple data segments, but typically it has at least one code segment and one data segment. So you can assume that the user program have at least two. And at most how many? We don't know, but at least two. So these two corresponding to the code and data segment. So MVM doesn't provide heap region to the user address space at all. So if some user called malloc, that will fail because there's no heap to allocate memory from, right? So this is something you want to discuss in your design document. How do you support variable number of regions? How do you support the heap? Here you can treat the heap as just another segment where the initial size is zero. You don't have any heap. Whenever you receive a spread call, you just expand the heap. This is another variable size region you want to support. And about the stack. How many, how do you manage the stack? Remember that a stack is also a variable size region inside user address space, right? And you want to improve upon this. So we want to provide variable number of regions. You do not want to put a limit on how many regions that user program have. And you also don't want to use one fixed stack size for all user programs, right? So this is the thing you need to consider when you design the virtual memory system. And finally, let's see how DanVM handles the address translation. And because DanVM handle design address space this way, it's very easy for DanVM to translate the virtual address, right? For each region, you have virtual base. You have physical base. You have the region size, right? For each region, you have three information. The base, virtual base, physical base, and size. So whenever you have a virtual address that you want DanVM to translate, first you find out which region that virtual address is. You can do it by comparing the vbase and impages, right? So first figure out which virtual regions that the virtual address falls into. Then you just figure out what's the offset inside the region. You first minus the virtual address by the base. So you got the offset inside the region. Then you plus that offset by the physical base. Then you get the physical address. So address translation for DanVM is quite simple, right? And this approach only works for DanVM because DanVM use this kind of scheme to manage all those physical and virtual addresses. Another limitation you will notice from here is that say I want to allocate four pages here, four virtual pages here. And conceptually, do I require these four physical pages that correspond to the virtual pages to be continuous? Because I have this virtual to physical translation, one flexibility we have is that for continuous virtual pages, we do not require the physical pages to be continuous. The physical pages can actually be scattered around all the physical memory. We have that translation that gives the user the illusion that these four pages are continuous, whereas in fact they are not. But here for DanVM, you can see that DanVM has to allocate these four pages continuously. Because it only has one kind of base plus size information for this whole region. Instead of each page, right? So when user want have a four page sized co-segment, that four virtual pages has to correspond to four continuous physical pages. That's the limitation you have for DanVM, right? How do you want to improve upon this? Well, do not use such kind of brittle translations. Use a clever way like page table, where for each physical page or for each virtual page, you have a translation, right? So this is where you want to discuss the design of your page table. What kind of page table you want to use? You can review the types of page tables we discussed in lecture, decide what kind of page table you want to use. It can be a one level page table fixed sized, or it can be a two level page table, or you can even use a link list. For each page, you have a list entry. Every time you just query that list. So there are a variety kind of design options you can use in terms of page table. So finally, swiping. Of course, DanVM doesn't suppose swiping at all. When there's no memory, it just panic, right? And you want to add this functionality. For those of you who work alone, I don't know how many of you are there, you don't need to consider swiping. But for others who work in teams, you need to discuss swiping in the design development. So the strategy for Simon 3 of Aurora is that you want to start with the physical page management by first design the com app destructor and implement these two functions, right? Then you move on to the address space design where you design the address space structure and also your page table. And finally, you want to handle the swiping. So there, how many weeks are there for Simon 3? Like six, seven? How many? Yeah, actually Jeff was thinking about bring the deadline of the Simon 3 early. All that. I'm not sure, it's not final yet. I want to bring it before the final, but anyway. So you can assume you have seven weeks, right? So you kind of do some time allocation for the physical pages, I would say at most one week, two weeks tops. Then for the address space, you want to spend another two weeks on it. And finally, you want to do swiping. Swiping will take you roughly three weeks to finish. That's kind of the schedule you want to stick on in the remaining of the semester. And for the design document, as I said, you want to discuss this stuff like the data structure design, what's the comment looks like, what's the address space looks like, and what's the page table looks like, what kind of options to use. And for each of the important functions, how do you want to handle it? For example, allocated key pages, free key pages, right? And that's for the physical page management. And for the user address space, you want to discuss whenever you have a VM file, whether it's a TIP file or a page file, how would you handle it? What's the overall flow of handling such a page file or TIP file, right? I finally want to discuss the AS family functions, function families, like especially AS copy and AS defined region. So those two I think is a little bit harder than others.  AS copy and defined region may be tricky to implement. And finally, you want to discuss, do you need synchronization? If you need synchronization, what kind of primitives do you want to use? Whether or not you want to use a lock to protect some shared resource or you want to use some other kind of synchronization primitives, but you need to consider in the whole process, for example, in the physical page management, clearly you need one synchronization where you want to synchronize the access to the com app, right? That's one example of synchronization you need to consider. And in VM thought, maybe you have other synchronization issues you want to consider. So that's basically what I got from today. Any questions on this? Yeah. What's that? No, I don't think so. The main pages are only for this course. Yeah, so I would suggest you read the code and read the comment in the header files. Basically header files is interface specification of all the functions. I think in the header file for each function they have one or two sentences to describe the functionality of their function. They don't have like what's the error code to return, that's that kind of stuff. They don't have that, but they do have some description. Any other questions? No, no, no, don't do that. So read the code of done VM. That's a good question. Don't do that because last previous year we have students do that and they start a lot in the later stage. At first it may seems easy to do. You have a lot of code to play with. You may want to tempt it to adapt it. Don't do that. So I would suggest that you guys start fresh from the clean slate. You can read the code of done VM and see how it does. As we have reviewed, there's not really much done VM does, right? So I would suggest you design the virtual memory system from scratch. First design a com app and then do allocated capabilities, free capabilities and so on. Don't try to adapt done VM. That will be done. Any other questions? Okay then. So I'll see you next time.


Thanks everyone for coming. I'm going to let you know that I'm getting over a super-packed cold. So if my voice does weird things or if I start hacking uncontrollably, that's why. So what we're talking about today is actually how we built and deployed Envoy at Lyft. And this is going to be a talk where I'm not trying to sell you something, we're actually going to talk about the practicalities of building something and deploying something very complicated into a very large existing infrastructure. I think it's a useful case study because we come to conferences like these and we learn about all the really amazing technology that people are building. But very often we don't talk about it in a construct that lets people understand you're coming from some existing technology and you want to adopt this new technology. That's a very hard problem. It's not as easy as saying we're going to run Envoy or run Istio or run Kubernetes. That's just not the way that it works. So first let's just do a brief refresher on Envoy as well as Service Mesh. There's I think 15 talks at this conference on Service Mesh, so I'm going to do this super briefly. I think everyone here probably has a good idea of what that is. The day behind Envoy is that the network should be transparent to applications. And when network and application problems do occur, they should be easy to determine what the source of the problem is. And what we find these days when people try to deploy microservice architectures is that the network is not transparent. It's in fact a huge thorn in people's sides. And it is very, very difficult to understand where problems are occurring. Is that problem in your application? Is it in the virtual infrastructure that you're using? Is it in some container runtime? It's almost impossible to know. And what I think a lot of companies actually find when they try to move from a monolith to a microservice architecture is they want the agility gains of actually moving towards that architecture. But then when they go to actually accomplish that architecture, when they go to actually accomplish that architecture, it becomes very problematic because people don't trust those services. And in fact, at Lyft, what we saw is that we had our monolith, and we'll actually all, you know what? It's fine. What we saw is at Lyft, you know, when we started to roll out our microservice architecture, we found that developers, they didn't trust the network. Like, they were seeing long tail problems, and they didn't understand how to debug them. They saw long P99 latency, and they would say, you know, we can't do this networking. We can't do this RPC, so we're just going to go ahead and we're going to write those features back in that monolith. So we see that problem where we want to move towards this architecture, but we don't have the tools to allow people to write business logic and not focus on all of the faults that are actually happening. So from a service mesh perspective, this is a very simple diagram of what the service mesh does. And the idea here is that we have a couple of different services. We have a sidecar proxy that is co-located next to every service. The service sends all of its requests or traffic to its local proxy. That proxy does all of the things that we all know and love around service discovery, low balancing, rate limiting, observability. And it's going to pick a target, it's going to send that request to the other side. The other side will deliver it to the service, and then it'll all come back. And the idea here is that each individual service only knows about its local proxy. It doesn't have to actually understand anything about the underlying network topology. That also allows it to be the case that your service can be written in any language. It doesn't matter what language it is, Haskell, Go, C++, Java, it doesn't matter. That service can talk to its proxy on local host. It can have something, some magic happen, timeouts, whatever. And then the response comes back and it comes back. And it really does make that network transparent in a way where people can focus on business logic and they don't have to worry about what the network topology actually looks like. So from an Envoy perspective, and I'm going to go through this super fast. Envoy is a proxy, it's a self-contained proxy, it is not a library. So it's an out of process architecture. So when I showed you that last slide, Envoy sits under the service and it basically receives all of those calls and those calls go out and then they actually come back. Envoy is written in C++11 so it's very fast as well, it's very productive. Fundamentally, Envoy is an L3, L4 filter architecture. So what that means is that bytes come in and bytes go out. So we can support multiple protocols. We can do REST, we can do Redis, we can do MongoDB, we can do MySQL. This is going to be problematic. And because obviously we do lots and lots of H1 and H2 and REST and all of these things, we do a lot of filtering up at L7. So we can operate on message boundaries, so headers, body, as well as trailers. Envoy is an H2 proxy first. So we can proxy from H1 to H2 and H2 back to H1. We spend a lot of time doing service discovery as well as active and passive health checking. So active health checking is the idea that I'm going to periodically send a ping or a health check request to some other service. Passive health checking is outlier detection. So that's the idea that I can monitor traffic that's happening within my data plane and I can go back, sorry folks. And I can go back and I can see if there's consecutive 500s or if a particular host has some type of success rate that's out of some variants. From an advanced load balancing perspective, we can do things like timeouts and retries and circuit breaking and shadowing and all types of fun stuff. And we'll talk about it a lot, but when I give talks about service mesh and Envoy, the most important thing is really observability. It's all about observability. It's about understanding what's going on in your system and if there are problems in your system being able to pinpoint where that problem actually happened. And that comes down to stats, logging, as well as tracing. The other thing that I would mention is that what you'll hear is that people talk a lot at conferences like these about service mesh. But all services, all internet services, typically have some type of edge proxy, right? And what you see today mostly is that people deploy different proxies for their internal proxies as well as for their edge. So people might use Nginx for their edge and they might use HAProxy or something else for their internal proxies. And from an operational perspective, 99% or more of what these proxies do are the same, right? They all do service discovery, they all do timeouts, they all do rate limiting, they all do circuit breaking. And from an ops perspective, running the same software on the edge and doing TLS termination and running the same proxy on your internal service to service network has huge operational gains. It's kind of like a core tenet of systems engineering. Less software you can run, less software you have to understand. It's vastly simpler. So this is Lyfts about five years ago. And this is almost a standard lamp stack, right? Like this is the way that every startup starts. We have clients, we go to the internet, we have the AWS ELB. Obviously now AWS has their advanced load balancer, but most cloud providers still have a fairly basic load balancing system. Lyft was originally built on a PHP Apache monolith and our single data store was MongoDB. And this is supposedly simple, there's no services. But what I'd like to point out really briefly here is that even in this basic architecture that we have here, there still is networking, right?  And even with this basic architecture, trying to understand what's actually going on here from, did the client have a problem on the internet?  Did PHP have a bug? Did MongoDB do something bad? It's actually really hard to know. So even in this very simple state at Lyft, we were already having a lot of operational problems trying to understand how is this thing working, how is it actually happening? And we're going to go into why we built Envoy and how we deployed it. But I actually want to show this because it's instructive again to see that you actually don't need hundreds of services to make something like Envoy useful, it can be useful with only very few services. So fast forward to today, this is Lyft's architecture today, roughly. We still have clients, we still have internet, we have a set of TCP load balancers, so very basic edge load balancers. And in there, we come to our front edge Envoy's. So we run a fleet of edge ingress proxies, which do L7 parsing and TLS termination and things like that. We still have our monolith five years later and probably for many more years. We are decomposing that monolith and we have lots and lots of services written in Go and written in Python. And we run Envoy, we have 100% coverage at Lyft. So Envoy runs co-located next to every service at Lyft and has hundreds of services. And then to our back end, we have MongoDB still, we have DynamoDB, we're actually starting to use Spanner from Google. And we obviously do a whole lot of stats and tracing and a whole bunch of other stuff. And again, the important takeaway here is that we run Envoy on each and every node. So every hop that we have from ingress to egress, we can have a point to point link from stats and tracing and logging that we can look at to actually understand what is going on. Okay, so that gives you kind of the overview of what happened. So like I was saying in the intro, you don't go from what we had before to what we actually have now. You don't do that overnight. You can't just automatically roll out Envoy on hundreds of services. It doesn't work that way. So what we're going to do is I'm actually going to talk you through kind of how and why we develop Envoy. And we're going to talk about the incremental steps that we took to actually roll it out and talk about the things that our developers at Lyft found most useful. Because as you're starting to think about rolling out some of these proxies, a lot of it is actually selling it to your internal teams. And a lot of times they want to know, what are we going to get out of this? Like is this actually useful? And the teaser here is that it wasn't easy. It was a lot of work. Okay, so what we did is we started with the Edge Proxy. And I actually talked about this briefly. And almost every internet application has some type of Edge Proxy. You have clients on phones. You have to get back to your back-end services. And from a fundamental perspective, if your app is not healthy at the Edge, it doesn't really matter what you do internally. So having that observability at that Edge location is super important. And we could have a whole long conversation about how it's important to extend that observability all the way out to the client, but that's not this talk. But from the Envoy perspective, it's critical that we get good stats out to the Edge.   They're black boxes where you don't actually understand what's going on. The observability coming out of them is still fairly bad. I think Amazon and their AOB product only got percentile latency metrics in the last six or nine months or something like that. It's quite bad.  They just tend to move a lot more slowly. So from the perspective of dashboards and tracing and logging, it was very important for us at Lyft to prove out this new proxy and actually put it on the Edge and replace the ELB and show that we can get very good stats. And what we found almost immediately, here's an example of one of the dashboards that we actually provide people at Lyft. And what you're seeing here is you're seeing on a per-host basis, here are our front edge nodes. And you're seeing stats like the number of connections per second, the number of requests per second. But then on the bottom row, you're seeing latency on a per-backend service basis, or you're seeing the 500s on a per-backend service basis. And this is a tiny portion of the metrics that we actually show people, but the idea here is that you go from a situation in which you're trying to go into S3 and pull down logs and kind of understand what's going on and the dashboards and CloudWatch are super bad to a situation where we have excellent visibility and we can much quicker pinpoint what are problematic services or what are causing problems for our customers. So again, it's all about observability, observability. So now we're on our web scale slide and if this was a 45-minute talk, we would stop here and we would watch the MongoDB's web scale video, but it's not 45 minutes. So I would encourage all of you if you have not seen the video and the little asterisk thing down there to go watch it right now because it is the funniest video that you have ever seen. So Lyft probably at this point is one of the biggest MongoDB installations in the world, I'm guessing. Like we have a lot of MongoDB and MongoDB has been a thorn in our side for a long time. And one of the things that you'll find with MongoDB even in the most recent releases is MongoDB has a very poor connection handling model and by that it uses a thread per connection in high-performance networking that is a big no-no. And what that means is that MongoDB will DOS itself very easily. If MongoDB does something bad, it'll start kind of refusing connections then clients will reconnect and then you'll have like a thousand threads and then the thing will just shut down and you're basically in this depth spiral. So one of the things that we realized that we could do pretty quickly with this proxy so we're already running it on the edge, now we have our PHP monolith. We realized that we could use Envoy actually on the PHP monolith to collapse connections from PHP to Mongo. So basically monitor and rate limit all of the connections that are coming into MongoDB from the client side of things. And hey, while we're here, we can actually parse the BSON and we can suck out amazing stats about latency and tables and query patterns and multi-gets and scatter gets and those types of things. And we can do this for all services so now we're also doing at this point, we're doing Mongo on Python. So instead of rewriting all of this rate limiting and all of this great functionality that we built in every Mongo driver, we write it once in Envoy. We point all of the applications at Envoy. Envoy talks to Mongo and lo and behold, we have no more Mongo outages due to Mongo Dev Spirals. So our current stack still looks exactly like this. So we now talk to Mongo from PHP, Python, and go all of that traffic runs through Envoy and via the Envoy filter chain that I talked about before, we can chain different functionality. We can build in stats, we can build in rate limiting, we can build in all of this cool functionality and we get that for free. The applications don't have to do anything. We just roll it out and it just works. So once we did that, so this is, you know, we're now going back a couple years and we have Envoy running at the edge. So once we have Envoy running on our monolith, we're still using internal load balancers from Amazon to actually do service discovery. So obviously when the edge has to talk to the backend service, we have to find those backend services. So services boots, they come up. They basically register into the load balancer. And then, you know, the load balancers is going to come and actually pick that back. And what you find is that, again, these load balancers, these cloud products, they're, you know, they're getting better, but they're still, they're black boxes. And that extra hop from an observant, from a kind of an operations perspective, when you have failure, now you have this giant thing in there that you don't know what the problem is and you have to figure out, well, it could have been a bad ELB instance. It could have been the EC2 network. It could have been my application. I have no idea. So you're putting this big thing that's in the traffic path, a single point of failure between any two endpoints. And that just makes understanding what's happening in that network topology so much more complicated. So what we found here is, but even with this, even with the ELB in the middle, because we're running Envoy on one side and we're running Envoy on the other side, we still get really amazing stats. We still get all of the egress stats from the edge nodes. We still get all of the egress stats on the PHP nodes. We can do things like buffer so that we can deal better with PHP, Apache traffic handling. So we're still getting lots of benefits. But the next stage here is that because of what I was saying before, we would like to get rid of those internal load balancers. Those internal load balancers, they make debugging harder. They're a single point of failure. You just do not want them. They don't serve any good purpose. So we would like to do direct connect. And at this point, obviously, if we're going to do direct connect, we need service discovery. And I'll take a little detour here. Is if you look around the industry, the way that most people at most kind of internet companies have done service discovery historically is they've done them using systems like ZooKeeper or Console or something along those lines. And these are great systems, they're fine. But what you'll find is that these are all fully consistent systems. They all use leader election. They all require care and maintenance, and they all go down a lot. And you'll find that most big companies have teams that are dedicated to keeping ZooKeeper and similar systems running, especially at scale. So when we started with Envoy, we made the kind of explicit decision that service discovery for networking, it's not a consistent problem. It's an eventually consistent problem. Systems scale up and scale down and things boot and die, right? It doesn't have to be consistent, it just has to converge. So we made the very explicit decision that we would go through and we would build an eventually consistent system. And that system, which we still use today, is about 200 lines of Python and a cron job. And the cron job runs on boot, basically. It runs every minute. We have a service that writes into DynamoDB and then we basically have a sweeper on that table, where if a service doesn't check in in X minutes, say 10 minutes, we just delete it. The system is dead simple. We literally have not edited the code in over a year, and we have had zero outages. And because of all the caching, it's eventually consistent. But because we cache in the service, because we cache in Envoy, and because of this eventually consistent process of the cron jobs right into the system, we have run tests where we have deleted the entire DynamoTable from under the service. Everything keeps going. We obviously can't scale up, but everything is fine. We make the DynamoTable again. Everything starts checking in again, and we're totally fine. So I would encourage everyone to think very, very, very carefully about whether you really need a fully consistent system when eventual consistency would probably be fine. So let's talk a little bit about how developers at Lyft actually use Envoy. So at this point, we're basically building the mesh. We're not fully rolled out, but we're building this mesh. We're actually getting a lot of value from it. And I actually mentioned this this morning during a panel, but you'll hear a lot of service mesh talks, and people like me will get up here and say that the service mesh is magic. It's totally amazing, and it is really awesome. But there's a little thing that we don't tell you, and that thing is that in order to get full value, especially around tracing, you have to propagate context, and that is required. There's no way around that.  So what we did at Lyft is we wrote a very thin client in Python. We also have one now in Ingo.  It's very simple. What you're seeing here is an example of someone calling a service called Switchboard. The idea here is that they don't need to understand how Envoy works. They want to make a request to a service. They use a client. They say what the service name is. They may set some options that you don't see here around timeouts or retries or various other things. And then this client is about 100 lines of code, and it calls down. It knows what port Envoy is actually running on. It actually goes through, and it makes the call, and it basically gets the response, and it unpacks it. And most importantly, it propagates IDs. So if we know, for example, in Python, that we have, you know, an X request ID or an XOT span context header that's come in and it's on the greenlit local thing or whatever it is in Python, you know, we'll take that state and we'll propagate it to the other side. And that makes it really easy for developers in Go. We do the same thing with Go context. But the idea here, again, is that from a dev perspective, doing a network call is as easy as import library. Who do I want to call? Make a call. Get answer. Okay. So at this point, we have now rolled out Envoy. You know, we actually took the approach of go big and don't do the tail. And I find that from a roll out perspective is actually better because if you can run this thing and show value on your biggest services, there may be a 90% long tail of small services, but if the big service people are happy, it's a lot easier to get these small service people happy. And what we found is that people were quite happy. Like we are suddenly saying, oh, like we're not having all these problems because we do more intelligent load balancing or hey, when a problem does happen, we can actually debug it. Isn't that awesome? So now what we did is we entered the slog period. And during this period, we actually had a spreadsheet and the spreadsheet was updated on a daily basis. We had a program that would scrape out all the services and figure out are they using Envoy or are they not using Envoy? We would build a Google spreadsheet every day and we would go through and we would burn that spreadsheet down. And that was a combination of just, you know, carrot and saying like, could you please do this? It'll work better, right? Like wouldn't these be great? We went in and did some of the work ourselves. To be perfectly honest with you, I was expecting to have this take much more of a stick approach. Like I thought we were going to have to go get in there and kind of get management to go make this happen. To be honest, it was a lot of carrot. Like people loved what they were getting. Like they loved the features. They loved the observability. They loved the fact that they just no longer had to worry about how certain portions of the system actually worked. So in hindsight, it was a long slog. But I think what you'll find is that most developers, most people that write business logic that actually run on a service mesh, they don't want to go back. And in fact it's been really interesting because Lyft when I joined was about 80 developers. It was quite small. We're over 500 now. And when I joined, there was people that knew the old way and there was people that knew the new way. But now people come in and they only know the new way. And I think for them it's kind of interesting to see because they just expect it to just work, right? They don't have to worry about all the problems that we actually used to have. So let's briefly talk about kind of how we did config management. And this again, I just wanted to explain, you know, that we come to the conferences and again, people talk about all this awesome stuff that we have. Things happen in a much less awesome way. Like not so awesome. So when we did Envoy, you know, what we did originally is we literally would hand write all the configs and we would check them in. And we would bundle them with the binary and we would deploy it out and that was how we did it. And that allowed us to move very fast because we could make breaking config changes. But it was very tedious because we were literally handwriting configs for each and every service. So that was obviously not going to scale. So what we did then is what everyone does, they bring in some templating code and we brought in Jinja. And what we actually developed is a system which we called config gen, which we still use to this day, certain parts of. And what that is, is you basically realize that a lot of the Envoy configurations, they become the same. Like there's a lot of commonalities in how they're doing it. It's mostly the same with a few variables that end up being different. So we ended up building a whole system that would effectively take a set of templates, a set of services and a set of variables and use Jinja, iterate over them and build an entire set of configurations. And it's at this point that we really realize with Envoy that in any complex Envoy deployment the configuration is going to be handwritten, or sorry, it's going to be machine generated somehow. There is just no way around that. You can do it using templating, you can use it with something like Istio Pilots, but something is going to do it. No one is going to sit around and hand make configurations for hundreds of different services. It's just not realistic. So at that point what we did is we moved to a hybrid system and the way that the hybrid system actually works is we did a combination of build time template generation and then what we would do is we'd actually ship the templates to each box and the Jinja files and we would actually run the template generation code on the host. So what we do at Lyft is every service at Lyft has a manifest and the manifest says things like these are the tests that run and for networking, these are the services we talk to, these are the circuit breaking settings, blah, blah, blah, blah. So what we did is we would take in this manifest data on the node, take it in with the Jinja templates and generate the final Envoy configuration on the node. So now we have a mix of central template generation with deploy but then the manifest information gets joined on the host during salt run because as you can see from here we're not yet a Kubernetes native shop. We're still using VMs and salt and all types of legacy infrastructure. So one of the goals of Envoy has always been to be kind of what I call a universal data plane. The idea is that we have a set of APIs, we can build a control plane and the control plane can speak those APIs and then Envoy will enact those configurations and it'll do it dynamically. And what that will allow you to do theoretically is have a single bootstrap configuration and a management server which provides the Envoy's all of the configuration. So you're kind of now seeing this transition period. We've gone from hand configurations to machine generated Jinja but now we're moving towards very complex management servers and we see that with Istio Pilot and there's actually people that are writing all types of different solutions. And in the V1 API we have a couple of different APIs. We have Service Discovery Service, Cluster Discovery Service, Route Discovery Service, Listener Discovery Service. The idea here is that ultimately we allow every aspect of Envoy that would be reasonably reconfigured to be configured by a remote management server. You're not required to use these so a goal of Envoy is that you take the complexity that you need. If you want handwritten configs, use handwritten configs. If you want SDS only, use SDS only. If you want the whole enchilada, use it. And at Lyft currently we use SDS, RDS, and CDS. We'll be using LDS soon. And this is roughly what it looks like today. And what you're going to see in all of the other talks that you'll attend here is that this looks a lot like Istio Pilot which is not very surprising because these things have evolved in parallel. And the way that this works is that in Envoy today at Lyft we have our legacy 100 line Python service discovery thing. We have a new service called Envoy Manager which is like Pilot. And we are now taking in all this information from our legacy infrastructure, from our cron jobs, from our manifests, from our merging them together and setting that configuration to Envoy's. So over time the config that we ship to every host is getting smaller and smaller until it becomes a unified bootstrap config that runs on every host, never changes, and everything else comes from those management servers. So that's that progression. But the idea is to understand that you can take still any of this complexity from end to end and you can build very cool systems on that. And we're actually now beginning our Kubernetes migration. So this will get a lot more complicated as we have our legacy infrastructure and then bring in Kubernetes and juggle all of it. So that will be very exciting. So thank you very much. Thank you for dealing with my voice. It's very shaky. You can talk to me on Twitter. I think I'm almost out of time, might have time for one question or so or two questions. My plug, Lyft is hiring. We have lots of good jobs. So if you want to talk to me, please talk to me. And we love building the overall Envoy community. So please join us on Envoyproxy.io. So thank you and I'm happy to take questions. Could you speak up? Yeah. Yeah, so the question is do we do functions or Lambda? Currently no. I think that Lambda is or just generally functions and Envoy is going to be a very interesting thing over the next, you know, probably two to three years. Mainly because functions at Lambda, they require networking. And it's actually networking is way more complicated in that environment because things come up and come down very quickly. So the answer is no, not right now, but I think that's going to be a big area of investment. So the question is at Lyft, what does the job stack look like? There's actually, if you look at the monorama talks, there's a couple of talks, two talks from Lyft that go into that entire thing. But I'll briefly say that we use a company called Wavefront for stats. We use a company called LightStep for tracing. We run our own logging system, which is Elasticsearch and Kibana. One more question? Yeah. So the question is Unvoy has a system called Runtime, which is like a decider or a feature flag system. It's pretty common that many companies have these things. We just have a Git repo that basically every time it gets checked into, it gets sprayed out to all hosts. Unvoy does a file system watch. One more question? Yeah. If you wanted to run NY at the service layer and officially provide that to the media. So we do run Unvoy at the service layer. We don't run it next to Mongo. We run it on the service side. Yeah, so yes. Yep. Exactly. All right, last one. Yeah. So the question is will Lyft move more towards Istio? The answer is absolutely yes. We obviously don't want to move more longer than there's Istio. And that's kind of what I was talking about before is that it's one thing if you're like doing this from scratch, but we have a giant system to migrate like hundreds of services, millions of requests per second. It's not something that we can just basically move. So the goal for us is going to be to move closer to pilot. I don't know at what time we would use Mixer. Components are actually separate. So one can use Pilot, but not Mixer. One can use Kubernetes Ingress, but not something else. So yes, the goal is to move towards it. Absolutely. Sorry, I have to go, but I will stand outside. So thank you.


Hey guys, it's Nate, aka The Foot Account, and welcome back to the channel. It is a promo Friday, and today we find out if EA are going to boost the power curve today with Fantasy FC and Fantasy FC heroes. I want to take a look at all of the leaks, and some of them are hot off the press, including some official stats for one player that is going to be coming in the Fantasy promo today. And what other content could EA release? We haven't had an evolution all week long with the Future Star Academy evos taking place there. What's going to happen there today? What about SBCs? What about the markets? If things go down how we think they might, we're going to talk about that all in today's video. If you're excited for it, drop a thumbs up. Subscribe if you're new. Let's quickly go over yesterday's content. The last full day of Future Stars actually ended with a W. Now, of course, we start off with our usual Thursday of Marquis matchups. Actually, one of the, I think, not as good Marquis matchups for the packs that you get back. The main pack out at the front is a rare electrum. I think it was a mega pack last week. That's better in my eyes, but I mean, that's just some couple of tradeable packs you can do during the week. As always, Marquis matchups is there. This is the SBC, Destiny Yudogi. We kind of knew that he was coming, right? He was the last day that possibly he could be released, and he was. And he went boom straight into my team, especially with the price of the SBC. 85,000 coins, 77% upvoted for this Yudogi card. It is a nice card. And I would show you the card. It's actually already in my starting 11, and it's already in the process of being evolved. I did the SBC, and then I have now started the evolution as well, right? A lot of people are doing this. They're taking this Yudogi card, and they are going and upgrading it with the 75,000 coin Evo. So basically, it's an 85,000 coin SBC, which you can then put into a 75,000 coin Evo. And you're getting yourself a left-back card that is pretty crazy. I'll be completely honest. Pingpass Plus and Jockey Plus as well. He's got long ball, whipped pass, block, and quick step with really some pretty good stats. Now, some people are asking, Nate, is this better than Alfonso Davies? To me, I would say no. He's not as good as Alfonso Davies. He's way more defensive, and I think I like that. Alfonso Davies runs out of stamina. This could be a great card at the end of the game. If you did it for the low price, you start Davies. And then sub on Yudogi later. Or if you just want to start Yudogi for a little bit more of a defensive left-back, that still can get forward. I mean, he has 81 shooting on this card. He has high, high work rates. And he does stay back more. I played a bunch of games with him yesterday, at least five, I guess. He stays back better than Davies does. A lot of people are asking, is he better than Havertz? And I think he's more defensive than Havertz, but you don't have the whip to pass plus. He still has regular whip plus, so it's very good. Guys, there's a lot of hype for this SBC card. A ton of people are doing this. It's got 2,000 upvotes, a 77% thumbs up on the SBC. I think people like this because it's a new prem left-back with the Italian links, too, recently with, you know, a Baggio and kind of viral. That helps him as well. But it's actually really well priced. Like, you know, we talk about prem tax when it comes to these sorts of SBCs, right? Where they usually overpriced the cards and they know people are really going to want to get excited about. And they didn't overpriced Yudogi at all. So that was very, very nice. And he does look cracked after the Evo. So that was actually a surprise yesterday. Another reason that you can tell people are doing that SBC like crazy is the fact that left-backs on the game have absolutely gotten demolished. I did not even notice how much bomb pastoral was down from 227K to 150,000 coins. That is a huge, huge price drop on the middle tier of the left-back market. Karchawi as well as another one. She went from 140K down to 100,000 coins. Zenchenko, the radioactive card I know is a very popular lower priced prem left-back. He went from 76K down to 57,000 coins. Left-backs across the game are in the mud because of Yudogi. That is a very, very solid SBC. So that's kind of the content that dropped yesterday. We do have some road to the final action to talk about because I think today, Friday, should be the day that they upgrade some of these cards, including Romelu Lukaku, Roma won on penalties to go to basically to qualify from the round of 16s. And that's how it was yesterday for Kowatas, Neres, Leao and Lukaku. They are all getting upgraded and they will join the list of players from the Champions League games earlier on this week. They will be getting upgraded sometime today. I don't remember when EA upgraded the live cards last week on Friday. I think it was in the morning. It was before content. I think they might do that again today. They could also do it after content. I think the rules are just sometime on Fridays. After the week's games are over, these cards will get upgraded. And yes, before you ask, David Neres, Boscagli from early in the week, they're going to be getting plus two overall because they are 86 rated. If you have those cards in your club, everything above 86 is just a plus one. But it's a plus one inform upgrade and the new inform system is 86 goes to an 88. So watch out for that today. Maybe a couple of price fluctuations there, but man, like some of these prices on the road to the finals continue just to be absolutely crazy. Like this Leao card is five mil, 4.9. 4.9 million coins. He's going to a 92 rated card. You're going to get some plus ones today. But that is absolutely wild how expensive he is. So that's kind of all the content that I expect to see today on this game. Let's get into the big news, okay? Sorry, that was from yesterday. Content in terms of the road to the finals. We got to talk about the content today. I'm actually still really excited for it, guys. I know yesterday we kind of talked about it. I know a lot of you guys are still down on it. It's really a gamble. Again, it's a prediction that I have that these cards are going to be juiced. We have some official stats that I want to look at that they look okay. It looks okay. We'll have to see. It does always as every promo Friday, it comes down to the pack weight, comes out of the prices of players on the market. And of course, how cracked the cards are because of the future stars, some of them are decent, some of them are fodder, right? But the pack weight was high and that was nice. But these cards are live. Let's talk about Fantasy FC and also do a little bit more background on this promo because we didn't do a lot of that earlier on this week. This is one of the biggest leaks I think we had yesterday. Griezmann, big name player. Imagine this Griezmann card with two play style pluses. I mean, it's basically a guarantee that a top tier card in this promo like Griezmann is going to have a two play style plus addition to it. I'm really excited to see what his card looks like today. Official stats with the play styles, right? Everybody uses gold card and maybe some of his earlier on cards this year with the Finesse plus and it was very popular. I think Finesse plus is still popular, but what else will EA give him? Maybe they'll give him technical and Finesse. That might be a little bit too good, but you don't know, right? Here's the official leak stats that we have right now. This is for Furlan Mendy. Furlan Mendy got a pretty sizable upgrade guys. I know that it's not 100% like a Fonto Davies level.  Oh, that's a little bit low. Passing, drilling, a little bit low. That is a Holit gang left back with 98 pace and 91 physical. And it's Furlan Mendy. He is always OP. And that upgrade over his dynasty's card, it's not just a simple plus one. It's a plus two overall. He got plus five shooting, plus two pace, and then ones, twos, and threes around the rest of the card. But the fact that he got Quick Step Plus is really intriguing to me as well. The Quick Step Plus on this Mendy card that he does not have on his regular item. He's got Rapid, Block, Flare. And now you're telling me he's gonna have Sly Tackle and Quick Step Plus, and I would not be surprised if they added another play style, a regular one, or two, or three to this card. Again, that's a good sign to me. With that plus five shooting, plus two pace, and still the fact that this Furlan Mendy could if he hits all the upgrade points, go up to a 93 rated card, albeit probably in like a month or a month and a half, it's gonna take for him to get upgraded to that level. That's a pretty sick card. And I think that's a pretty sizable boost. I know this dynasty's card has been out for a while, like it's been out since December. But I think that's a pretty sizable boost, and that gives me hope for the rest of this team that it could be pretty cracked. Now, speaking of upgrades once more, we talked about the upgrade path and how it works. Last year, the cards did not get upgraded until two weeks after the promo team came out. I don't know if you guys remember, we were really disappointed last year because these cards came out and they didn't get upgraded for a couple of weeks. Like literally they came out on March 3rd, and it was March 17th when the games first counted. I believe that this year it's different and it's way more hype because the upgrades start today. Like if a club has a game today, which there is one leaked guy, one leaked card who actually has a game today on Friday, I think he has a chance to score a goal, assist a goal up here in a match and start working on his way towards those upgrades. So we're going to talk about that, but I think that's going to be a big part of today's promo as well. It's almost like ones to watch. We didn't have ones to watch earlier on this year, and it's almost like that because the upgrades happen basically right away as there's a lot of games this weekend. So I think the upgrades start today, which is very, very hype. Now let's go over the rest of the leaked cards. We've looked at Delict, of course, and then the Konate is a new one that we have not looked at yet. I mean, everybody just did either Bade or DeSasi, right? A lot of people were hyped about DeSasi, maybe even Canovaro or Rycard, a lot of center back SBCs recently. Looks like we're having another center back to join the mix, not as an SBC of course, but in packs, Konate, his card's probably going to be pretty juiced. These are predicted stats once again. Will EA give out Anticipate Plus for this Konate card, or will they hold it back? We'll see. I mean, if they're going all out, they would give it out. Then there's the Griezmann, of course, that we looked at. Same thing with Marquinhos. I mean, guys, when I look at the players, the quality of players in this promo, you might be like, ah, Nate, reused cards, like for Marquinhos and for Griezmann and all that, and I feel you. But these are the promos that big name players get into that actually bring a lot of hype to the game, just because it is the big name players, even though we sometimes get frustrated about them overusing stuff and reusing cards and all that. Marquinhos has had a Trailblazers this year and I think that's about it. So this could be a huge upgrade for him, and of course, again, a lab card with potential upgrades. We've got Jack Grealish, so that prediction earlier on in the week about how the card's upgrade was actually a leak as well. Grealish is coming as a fantasy card. These are predicted stats for all of these. The thing to think about with some of these is, especially with Grealish, injuries. Injuries are going to play a big part as we kind of calculate how much a card is worth because of how many upgrades they could get and how much hype there is around a card. I believe Grealish maybe was just subbed off injured recently. I'm not sure, same thing with Griezmann. So that's a big thing to look at with these cards because injuries could really kill the hype on an upgrade, even though the upgrades take time. If they're not healthy, that means they're not even eligible to hit some of the upgrades and a part of the pass. So that's something to keep consider with these. And this is Lacazette. Lacazette's coming as a fantasy foot and he actually has a game today. I believe Leone play in League One today. So I don't know, this card looks disgustingly good. Like if that's the Lacazette that they released, that is going to be a nuts, nuts card with two playstyle pluses again. We hope for basically all of these stats are predicted but that would be cracked if they do that. So that's hype, right? And he would probably be a middle to lower tier version. Part of the promo team as well you'd think, right? I mean Lacazette, big name player for sure but Grealish, Marquinhos, Griezmann, some of those guys are going to be pretty high rated for sure. Now let's look at some of the heroes because the Leagues look pretty solid guys. There's just not that many Leagues so far. Like only one, two, three, four, five, six, seven. Like maybe seven or eight players if I'm missing one or two. That's not very many players to be completely honest for who's leaked in this promo so far. I think there's going to be some surprises today. Especially as we start to get more official stats, leaked stats being tweeted out and some stuff like that. Let's move on though to some of these heroes. We're first looking at the hero upgrade path. It's a lot shorter, right? If you guys remember from last year, they don't have as many opportunities to upgrade because they are heroes, right? And EA wants to keep releasing special versions of heroes throughout the year, you know, for all that. So the base card's going to get an upgrade and then the club winning two games will get the first upgrade for the hero and the club scoring the 11 goals in four games is like the potential plus two for the hero card. So I feel like most of the heroes should be able to get a plus one but it's going to be difficult to get a plus two for sure because that is the hardest upgrade to get out of all this. A club scoring 11 goals in next four league matches is pretty difficult but each of these heroes will be represented by a team, by a club and it is company represented by Manchester City. So he's getting another card today. He's going to be in packs today for a Manchester City upgrading hero card. That'll be one of the better center backs in the game too. His cards have been pretty, pretty good this year. Now let's take a look at a few more leaked heroes. We've got Tevez. So I'm guessing if company is city, then Tevez is probably going to be united if he's going to be a Manchester United or a Premier League Tevez. I think that would be the most hype especially for a lot of us on this game and that would make this Tevez the most high version. We just had a Tevez SBC but now we're getting one in packs as well that will be live and potentially upgrading. We've got Ruby Costa who needs a definite upgrade from his card. Even the UCL version. Nice little upgrade there for him and he could be usable. This is one of the things I'm a little excited about with this promo today. Hopefully they give some of these heroes two placed out plusses and some pretty solid juices as well. Oh yeah, we came up with a new word yesterday. I say boosted and juiced all the time. So we came up with a new word in the Twitch stream. Juiced. So we're going to say we're hoping for these cards to be juiced today. Marquisio is again, another card that is going to be in their promo team today in packs. I don't know if any of these leaks have to do with the SBCs because there are two heroes that are going to be coming out as SBCs this next week. Kind of like the icons during future stars. The heroes this next week during fantasy. But we'll see. We don't know who those are yet. We've got Paldo Foutre is going to be coming as a hero card as well. His pace and dribbling is going to be insane. And maybe now hopefully the shooting in the passing can start to catch up because on his base card they were like really low. He had insane dribbling and pace and his shooting and passing is what let you down. Well, hopefully now with two placed out pluses as well, technical plus has to be one of them because that's like his thing. Maybe they give him trickster. That wouldn't be as high. Hopefully two really good placed out pluses for that card and he could be a beast. Same thing with Kessler. Official stats, dynamic and play style. So Kessler going to a 91. I did not see this. Let's pull up Kessler's UCL card. 90 rated to a 91. I guess they can't give the heroes that big of a boost like they could have Ferlan Mendy. She's getting Trivella and Energizer plus. Oh my gosh. 90 shooting, 91 passing, 91 dribbling and 86 pace with 81 defense and physical. What is that upgraded from this glass card? Plus three shooting? Wow, plus two dribbling, two pace, one overall. Again, guys, these official stats and the dynamics and the play styles, I think they're juicing these cards today, man. That Mendy looks good. This is a bigger boost on a Kessler card than I would have expected. This looks pretty good, man. Yeah, I'm getting excited for this. I mean, I don't think I can't stop myself from getting excited, bros, because these upgrades look pretty sizable. So I'm really curious to see how that transfers forward during the rest of the team getting released today on this game. One thing to mention is this has been a two week promo in the past historically. So I guess there's a chance. So you're telling me there's a chance? Yes, I am. There's a chance that some of these cards maybe are not actually released today. And there could be some that are released a little bit later next week. We'll just have to see. Now, one thing I do want to talk about as my SBCs do not want to load in. This is really weird looking. Okay, now it's fixed. We are getting an SBC player today. The only SBC that we have leaked as of right now is Teemo Burner the Burner Spurs Transferred card. I'm excited for this one. If we get back to back days of Spurs SBCs, that would seem unlike EA. I'm just being completely honest.  Also, if you need a Pacy, permanently get tackered. If they were to give him this card, we're talking 96 pace, 87 shooting and 88 dribbling. That will be kind of sick as well. Yeah, I don't know. This one could be one that comes out like hemp last week who you know is going to be an OP card. Teemo Burner always has overpowered cards in this game. Just with his build, his pace, his dribbling. It's going to suit this meta for sure. Left stick dribbling, technical plus if he has that, right, he could be insane. I could see it being like hemp where it's a little bit overpriced, but it's a meta card. So people are willing to pay for it. The thing is Spurs upgrades, even as a Spurs fan, it's definitely not a guarantee that in the next four league matches, he could get even a plus two or a plus three. It's possible, but you never know. And that's some of the fun and some of the risk and the hype with these cards for sure. But I think that's our SBC player today. Again, there could be more leaks as we head into content today that could change some of those things. I just don't see an SBC player today being cheap. That's the thing that I'm trying to get over to you guys. It's like, cloudy opinion is expiring today. That was a cheap SBC for the quality of player that we were getting two weeks ago during Future Stars. I don't see one being really cost effective to the cost efficient because does EA wanna give out a really cracked card that also upgrades? Like usually for live cards, they're not cheap in terms of the SBCs. So, I mean, Kimmich was decent value and Lautaro was decent value, but they weren't like really, really low cost. That's what I'm trying to get across. So I don't know how the SBCs are gonna look today, guys. I mean, one thing we do know for sure is that the 86 plus team of the group stage is refreshing and the team of the week player pick goes away. So you expect that to refresh as well today as it always does every single Friday. But a lot of stuff's going away. The 85 plus player pick, which I gotta get done two more times. The icon pick does not refresh today. It refreshes tomorrow on Saturday. And the 82 times 20 goes away. I have that pack saved that will open up for the fantasy promo today too. But that SBC is going away. This one is refreshing. And I don't know, guys, I feel like there's some sort of surprise coming with content today. I just don't know what it's gonna be or where it's gonna be. Objectives, we need XP, okay, EA Sports. We need XP badly. Give us theme team pursuit or something to give out some XP, man. Because a lot of people would like to get to level 10 and get Eunice Musa. And if they don't drop XP today, then you're gonna run out of time to put this Musa into the glow up evolution because it is expiring soon. Or entering into it and putting a player into it is expiring soon. One thing I will mention is we probably get an objective player today for this promo. Again, objective players are always pretty easy to get done, but they're not as juiced, right? Our new word, back in action. So watch out for that today. And then we gotta talk about evos because we have not had an evolution for this entire last week. It's all been, well, since Sunday, right? When we had the glow up. The last week has been pretty quiet on evos. It's all been Future Star Academy. Future Star Academy and all that over the past couple of weeks. There's only one page of evos on this game. Some of you guys probably even have less because you've done most of these, right? Yeah, there's gotta be an evo today, but I do not know what it is. There are no leaks as of right now as to what an evo coming today would be as well. I can't imagine that they would want to give out a player that would be a fantasy player. I don't think they'll let us customize one of those because we just had a promo that included a special car design for the evos. And I think this will be an off promo where we just kind of get some more generic evos. Upgrading hopefully cards that we've already had in our ultimate teams. Like an evo kind of like this, paid and also free. Like I've got a bunch of cards that are, you know, 83 to 85 rated that are just evo cards in my club that I would love to upgrade and get to a higher rating and a higher level. Hopefully with the power curve going up, if it does. Right, that's one thing with these stats looking as juiced as they are on some of the official stats, these promo cards if the power curve is truly going up today, then hopefully the evos will be following suit. But watch out for leaks for those because again, it's just been all academy stuff right now. Now, before we get into some market talk let's look at some of the packs that are coming this today and this weekend. Here's the pack or the player pick that interests me the most. There are some pretty crazy packs in here but this is the one that's kind of crazy. Heroic halftime player pick. It includes one UCL triple threat or fantasy hero player pick rated 88 or higher. That's kind of crazy. That's the first hero player pick that does not include base cards but it's being added into the game as a pack. That sounds like a store pack 100% by the name of it and everything but that includes also current in packs fantasy heroes. So that's really interesting. I don't know when that's gonna be dropped. I maybe wouldn't expect that until team two. Maybe if it's a one week promo we get it sometime this weekend we'll see. There's also another icon pack which is including three base icons with one 88 plus. Also 50 rare gold players rated 80 or higher and then we get into some of the big store packs. The supreme halftime on core pack. It's again one of those that is giving out some old promo versions. Radioactive dynasty's group stage versus winter wildcards. Three of them are guaranteed to be 88 or above. That's a pretty crazy store pack and then also an 84 times 20. That's gonna be like 6, 700,000 coins. That's gonna be very expensive. So that's a massive pack there that is upcoming but let's talk about the market a little bit because we already just mentioned it and I wanna talk about it some more. The power curve, right? And that's a big conversation with how these cards potentially getting upgraded could impact the market and we touched on it yesterday for sure but I think you could see more impact on that today. It's kind of gonna go I think one of two ways. Way number one is the cards that they dropped today most of them actually are juiced. They are pretty crazy. We get a decent SBC, we get a decent objective, maybe a decent EVO as well and we get more and more cards that are double play style plus and of course the new cards, the fantasy cards that are coming out are live in that way to me, I would think that there would be a little bit of a correction to the market, especially maybe on some of the team of the years. I mean, think about it. If you're somebody who has team of the year, I don't know, let's say like team of the year Sam Kerr or team of the year Sophia Smith and you've used this card now for a couple months, you know, team of the year it's come, it's gone. These cards were super elite when they first came out but now, you know, they're still good. I mean, let's not say anything bad about team of the years. They are still incredible. Like how many cards stack up against the Sam Kerr or Sophia Smith as team of the year strikers? Not very many, but also they've started to slowly drop down in price as we've gotten more and more double play style plus cards that at least kind of compete with some of these and you see that Griezmann? I mean, there's gotta be some people on the game today that look at their Sophia Smith and like, okay, I've used this card now for a couple weeks. Yeah, it's freaking good but I really wanna try out a brand new cracked Griezmann card today, you know? And it all comes down to the price of the new cards today. I don't think they're gonna be cheap. Like with boosts and with ratings like that, EA is not gonna wanna give them out like candy, like super, you know, super supplied like we had with Future Stars but I mean, if they're anywhere near affordable today, like if this Mendy is 1.5 million coins, I think everybody's expecting them to be two mil plus because this Mendy last week was 1.5 and this one is a plus two with plus five shooting plus two pace, two play style pluses and it's alive and upgrading. I don't think these cards are gonna be cheap. That's not, I'm not trying to tell you that. I'm trying to tell you that some of the other cards that are pretty expensive, the middle to high tier in this game, we're just trending towards that time of the year where there's another market correction. It might not be today, it might not happen today but all I'm saying is, if these cards are actually all truly boosted like the two official stats cards that we have seen so far look like, there could be a slight bit of panic on the market and I'm not saying it's gonna be like, oh my gosh, this Sophia Smith's gonna go from 2.6 million coins down to 1.5, you know, it's not gonna be anything like that. If you bought a team of the year when they were near their lowest or even if you bought a Sophia Smith for like two mil flat, I think you're chilling. You know, like one, two, 300,000 coin drop on a two, three million coin card is kind of what I would expect, that sort of thing. Maybe for a card that's like a million coins or, you know, maybe an example is like Basha. We get the new Furlan Mendy today. If Furlan Mendy is 1.5 or basically if Furlan Mendy is under two million coins, I think that hurts this card. I'm gonna keep it a buck 50 with you because yes, Basha has 90 passing but Furlan Mendy also is just Furlan Mendy with 98 pace and some people who wanna pay for the biggest best French left back in the game, I think you're gonna have to make a choice. Do they go for Basha or do they go for their brand new Mendy? You know what I'm saying? So statistically a lot of these cards are gonna be really good today just because of the boost that we've seen but it's gonna come down to the play styles and the pack weight and the price if they're gonna make a market impact today. Maybe the market impact isn't seen for the next two or three weeks or maybe even just later on this weekend but yeah guys, I would say again just another slight warning if you have expensive, tradable cards with double play style pluses even some of the live cards just be careful with them, all right? And if you're really worried about it maybe take the cash you can always buy back at a later date maybe in a week or two if you wanna be safer but usually late February, sometime during March there is a market crash whether it's before a foot birthday during or after there's always a market crash in March and I just think that we're getting towards that time of the year where EA is gonna try to do some big things again push the power curve in the cycle that we're in right now and today might be the start of that so just be careful with the market I think that other places in the market today could end up doing good if all the cards are expensive then what's gonna go up? I mean not everybody has millions and millions of coins, right? What about the lower tier? What about Hoyland? Hoyland is one of my favorite cards I think out of this promo if there's not a cheap striker today that is released in the promo team Hoyland could very easily go up he's one of the most hyped cards in this team you've already seen Olise go up a lot Rico Lewis is rising some Xerxes up a little bit Mucoco's been rising Cold Palmer is up a little bit too Futurestar icons David Beckham is really hyped he's gonna go up most likely Smith has already rose 30,000 coins Rivaldo is up even Gattuso is up a little bit so kind of like last week if the new cards that come out today are a little more expensive just like Futurestar's team two was a little more expensive than team one Bonaface went up right last week you remember when I was flipping him Elliot started to go up last week I think there's some of that potential today with the Futurestar some team two Hoyland I think would be one of my favorites because he's still low-ish so yeah I'd watch Hoyland it would have to be somebody with two placed out pluses I think Olise is already gone a lot of these other ones are a little more expensive for my liking so yeah I'm not really investing today I'm gonna keep my coins pretty liquid I think for the most part I do have Suley still and Suley's getting upgraded today so I'm excited for this card to go to a 89 rated what's his price right now he's like 350, 360 but this card going to 89 today there's probably gonna be a Dortmund player in this promo team I'm pretty sure we haven't seen one leak yet but there's probably gonna be one I know we're getting delict today as well but delict is gonna be expensive with the buying links even though they're not playing good so yeah that's kind of what I'm expecting there for the market it's gonna be a jumbled mess today and it's really gonna depend upon what content is really released I don't think it's gonna be like a crazy market crash day unless EA really come out and do something bananas the last thing to talk about is Fouter because Fouter had a really nice rise in that middle tier once again should be a reset last week I believe on Friday these cards had a bit of a drop as well unless there was a spike was there a spike on Friday? No, they didn't Thursday there were 5,000 coins for 85s they went all the way down to 4.3K in a Saturday and then they started to rise during the week as they had a lot of demand I would expect these to come back down in the 5,000 coin range 86s I mean you do have the 86 plus pick that is coming out today and that's an 85 rated squad with an inform so 85s, 86s could be in demand a little bit but I think over the course of today, tomorrow they could drop down a bit unless EA do something crazy they should drop a bit but then on Saturday you've got the 88 plus icon pick refreshing one last time and maybe that makes these cards rise up a little bit high rated Fouter I mean man there's nothing to make these guys go up there's really not there has to be an 88 or an 89 rated squad requirement to make any of these cards move 89s, 90s or 91s and as of right now it just doesn't look like anything that is going to be a well completed SBC is going to make that stuff move right now in this game so that's the video for today guys I got a couple random packs and objectives and the 82 times 20 saved up I got to do my 85 plus player picks before content today we'll save those for the new promo and yeah it might be an interesting day today I'm hopefully hopefully we get excited that's what I want to be excited again today I know that Future Stars had its good moments and all that but we're ready for some juiced power curve boosting cards I think most of us are even if they're expensive I think we're ready for it all right we're ready for the next move upwards in stats on these cards in this game so if you're excited for the fantasy promo today drop a thumbs up on a comment down below if you have any questions and of course subscribe if you're new I'll see you guys on the Twitch stream today that link is down below in the description it's been Nathan Ford Accountant see you guys there peace out