So these are, I will put these up on piazza as well, but these are the midterm statistics.  So come up to, well, now let's do them upstairs. So come up to 301B or that vicinity, and I'll give you your grades back, and maybe Aditya and Zehe will be there to help me. But yeah, I don't know.  So again, I'll put these up on piazza so you can kind of compare. Looks like, let's see here, I guess a chance he's my laser pointer. Oh. Yeah, question five seemed very, so these include some statistics from the alternate midterm that other people took, which were different questions, right? But there's not too much mixing in. So this seemed like an easy question. I don't remember which question that was, but it was popular. Most people got the multiple choice.  But we'll put these up on piazza, and you guys can pick up your exams later today after class. Cool. All right, so today we are going to talk about, we're going to go back a little bit to Monday and maybe go over some of that again because I just had one of those lectures Monday where I just didn't feel like I nailed it Monday. So we'll go back and we'll try it again. So and then we're going to talk a little bit about, now that we have this nice idea of using pages as a fixed size segment to translate addresses, how does the kernel, so we're going to talk about the interaction between hardware and software, talk a little bit more about hardware and what hardware does, but there's this other question of how does the kernel store these mappings, right? And how does the kernel look up them efficiently when the kernel needs to? And then we'll talk a little bit, if we're assuming we get to this today about ways that the kernel can reuse, underutilize memory by moving its contents to disk, right? And when we might do that and why that might be a good idea. And then what happens if a program tries to use memory that I've moved to disk behind its back, right? All right, so right midterms after class, come up to the 301B 303 area. I also just wanted to mention this because this is a time in the semester where perhaps you guys have started to actually work on some of the homework assignments. So if you're having troubles with your partner, please come talk to us. Please be proactive about doing this, right? The earlier you guys come to us and talk about struggles you're having with your partner, the sooner we can intervene, right? If that's an issue, right? The partner difficulties are really up to you guys to define, right? So you guys at some point hopefully will graduate from this wonderful institution and go off into the world to do, you know, let's be honest, probably less exciting but more profitable things. And in that context, you will have to work with other people and you will find many of those other people to be a challenge to work with, or at least some of them. And so you guys, the partner portion of this class is designed to give you guys a little bit of preparation for this type of environment. But if your expectation of your partnership is that you and your partner are going to each do exactly 50% of the work, that's probably maybe a little bit unrealistic, right? On the other hand, if you find yourself doing 95% of the work and are really excited about it because it's really awesome, then maybe don't come talk to us, right? So this is, again, this is really up to you guys to define, right? And this is also a time when it's okay not to use piazza. Although, I don't know, if you wanna post funny things about your partner problems on piazza, I guess that would be okay. Maybe use the anonymous student feature, right? Okay, any questions about this stuff? Okay, here's these numbers again. They are there. Uh-oh. Clicker confused by slides. Uh-oh, slides confused by slides. Like this doesn't. All right, so any questions about paging? About pages, fixed size segments. It simplify the process of address translation and reduce internal fragmentation. So we're gonna do a little bit of review here. So we had this, we've been talking about this all along without really stating this explicitly, but we've been playing, we've been using a trick here, a common systems trick that's used to make something slow look faster. What is that, what is that trick, Nick? Nick rhymes with trick and caching, right?  So the typical approach is, when you have something big that is slow, you put something smaller that is faster in front of it. So again, the L1 cache on your machine is very fast. Well registers are very fast, but there's only a few of them. The L1 cache is bigger, right? But it's a little bit slower. The L2 cache is even bigger and then potentially more slower. Memory is much bigger but slower and then your disk is huge and really slow, right? So you can think about operating systems as essentially managing a series of hardware caches, right? So what is the thing, but let's be more explicit.  What's the process here that's slow, Manish? When we're talking about virtual memory, what's the slow process that I'm trying to make faster? Okay, so that load and store is what causes me to have to do this, but AJ, what load and store causes the system to have to do what? Right, translate an address, right? So we talked about, and this would be equippling difficult to because you can't imagine how slow your system would be if every address translation had to be done by the kernel, right? So the kernel is too slow.  Sean, Paul. Yeah, TLB or really this memory management unit in general that we've been talking about, kind of this vaguely defined MMU, right? The idea is that the kernel wants to be able to tell the MMU how to do the translations and allow the MMU to perform most of the translations by itself, right? So what's slow here is the kernel's process of manually translating addresses and the cache I'm gonna put in front of it is this MMU. So I'm gonna tell the MMU about some address translations and I want the MMU to be able to do them going forward, right? And we just talked about this, right? Or we use this translation look aside buffer and that's the particular part of the MMU that's going to, we're gonna talk about caching these translations, right? So you think about the MMU as a component that not only includes this cache but also is able to process those virtual addresses, right? So the CPU sends a virtual address to the MMU and the MMU sends a physical address back, right? Or punts and says I need help, right? But we hope that that doesn't happen too often. How do entries get into the TLB? Where did the TLB entries come from? We can, it's good I'm a lip reader because I can't hear you but I think that you said the right thing. The operating system tells the TLB what to do, right? So again, this is classic case of separation of mechanism and policy, right? The mechanism here is the TLB and the MMU translating addresses. The policy is dictated by the kernel. That's how we want it, right? We want the kernel to decide how addresses get translated and allow hardware to translate them quickly, okay? So we talked a little bit about how hardware could translate base and bounds type addresses and segmentations and segments, right? Using segmentation. But then we found ourselves having this problem, right? Which is that as segment size has changed that would cause problems for the operating system in managing the memory on the system, right? Specifically the potential for both external and internal fragmentation, right? Internal fragmentation in terms of unused pockets within the segments themselves. And external fragmentation in terms of as segment size has changed trying to reallocate segments in different parts of physical memory, right? So what we did is we said, let's make things simpler for ourselves. We're gonna choose a fixed size for the segment, right? And we're gonna call these new fixed size segments pages. And there was this trade-off here, right? So as the page, so what happens when the page size gets really small? Let's say I had very, very small pages. What undesirable thing starts to happen? So I'm gonna argue that that is actually what happens when the pages get big. What happens when they get small? Dan, we tried the same answer twice and it was still not right the second time, yeah. Why? Okay, we're getting closer, right? So Bart, you wanna take us the rest of the way? As the segment size gets smaller, I'm gonna have to tell the MMU about more translations but why is that? Yeah, so the cache in the MMU is a fixed size, right? So you think about it, the amount of memory that the MMU can translate without asking the kernel for help is the size of the MMU's translation cache times the amount of memory that each translation covers, right? As page sizes get smaller, the amount of memory that's cached by the MMU, the amount of memory it can translate without asking for kernel help goes down, right? Does that make sense, everybody? As pages get bigger, it goes up, right? However, as several people pointed out, as pages get bigger, what's the problem? Dan, internal fragmentation, right? Why? Because a page is some big bundle of content, right? And the pages we're gonna talk about either has to be in memory or not in memory, right? So here's another question. Hopefully you guys are developing some intuition about this. Why can't, so let's say I had like a 32K page. Why can't I move half of the page contents to disk, right? Or why can't I only have, you know, say, oh, why? You know, why not? Why can't I have 16K of the page that's in memory and 16K of the page that's in memory? Jeremy. Well, but again, I'm gonna say, let's say I'm gonna create this new concept called a page fragment, right? Which is 4K and my pages are now 64K, right? And I'm claiming I'm getting the best of both worlds. I can move around these little pieces of the page, but why can't I actually do this? I don't know, we're getting warmer, right? So again, I'm claiming I don't have to move the whole thing, but what's the problem here, right? So let's say that I'm trying to implement this idea. I have my big 64K page, right? And the page is in memory because the process has used it, right? And there's a translation that is cached in the MMU, right? Oop, sorry. So the MMU is caching a translation for this page. So while, so here's a question. While the MMU is caching a translation for the page, does the operating system, does the operating system know whether or not, does the operating system know anything about what parts of the page the application is using? I see shaking of heads, why not? Tim, I tell the MMU, this is how to map this 64K of virtual address to physical addresses, right? What doesn't happen for the rest of the time that that translation is in the MMU? What don't we want to happen? Why did we tell the MMU about the translation in the first place? Sumit. Yeah, we don't want the kernel to know, right? The MMU is sitting there translating all these virtual to physical translations behind our back and the kernel has no clue, right? So because of this, there's no, so for example, if I wanted to take a piece of that page and move it somewhere else or take it out of memory, there's no way to stop the process from using it, right? Because the, so essentially what's happening here is my protection granularity is also a page size, right? And this is actually true when we go back to thinking about how we grant and enforce and revoke memory, right? It's all now done on a page size granularity, right? I cannot give a, I cannot give a process, if I have a 4K page, I cannot give a process permission to only access 1K of memory. It's impossible, right? Because I give it a mapping for 4K and it can use any of that 4K, right? Without me having any idea, right? So this is something to keep in mind. If I have 1K of physical memory that I want a process to use, there is no way to do that with 4K pages, right? I have to give it access to the 4K that includes that 1K. Does this make sense, everybody? Nod your heads vigorously. Head banging sort of thing. Okay. So pages, besides setting our protection boundaries, right? Pages also dictate the size of these kernel data structures we're gonna talk about today that are associated with memory mint, right? So as page sizes get big, the other nice thing in the kernel is I don't have to keep as much state around, because fundamentally there are fewer translations to translate a fixed amount of memory, right? So if I take a fixed amount of physical memory, the way I allow a process to use it is I cover it with pages, right? So as the pages size gets bigger, I have to use fewer pages to cover the same amount of physical memory. As it gets smaller, I have to use more. And we talked a little bit about address and execution locality, right? And how this helps us, right? So we talked about the fact that if we look at the memory addresses that are actually used by a process at runtime, they are temporarily clustered within the process's address space, right? So I'm fetching a bunch of instructions from this one part of memory, right? And maybe I'm also using another page over here for data or something, right? And I do that for a while and then I jump over here to do something else because you clicked a different button or whatever, right? But there's a lot of locality, which means that a small cache of address translations on the MMU, right? So a small TLB actually goes a long way, right? So small, and you guys will find this when you do assignment three, right? Because the TLB on your system is not very big, but you'll find that when you start a process running fairly quickly, especially for the processes on your system because they're really, really small, right? So it's very, very easy to cache all the translations that one of your processes needs in a TLB with fairly limited size, right? Okay, so we talked about 4K, which is the kind of the common page size. There's some larger systems that allow, you know, I think some of the new x86 architectures have these huge pages, which could be like 64K, right? But we can think of in this class as pages being a specific size, finish. Yeah, I mean, we think of the MMU as a part of the system that performs virtual to physical address translation, right? One of the things that it uses to do that is a cache of address translations, right? But there's also circuitry involved in just doing the translation itself. So when we talk about the process of taking a page, a virtual page, or translating it to a physical page, part of that is looking up the virtual to physical page number mapping in this cache, but there's other parts of that, right? And all that circuitry is all kind of wound up in what we call the MMU, right? And we'll talk today a little bit at the end if we have time and if not on Friday about other things that x86 MMUs do in particular, right? So x86 MMUs are a little bit more sophisticated because x86 MMUs will actually use kernel data structures to look up translations, right? Behind the kernels back, right? Which further limits the amount of work that the kernel has to do, but it also ends up having to impact on the amount of control that the kernel potentially has, right? So again, we think of pages as fixed size segments, right? Pages are like what? Fixed size segments, right? Review slide follows material fairly quick, right? And again, all the virtual addresses on a single virtual page map to the same physical page, right? These are fixed size segments. So when we talk, and that's why we can split pages into this page number, right? Which identifies the virtual page and an offset, right? The offset is like a segment. It's to both the virtual page and the physical page. And again, if I load a translation for a page, all the addresses are gonna map to the same physical page of memory and there's no way to protect anything within that page boundary, right? So I can't protect 32 bytes of memory with 4K pages. I can't even protect two kilobytes of memory with a 4K page, right? So when we do page translation, we split the page, we split the address like we do with the segment, right? Into two parts. There's a part that identifies the page number and there's a part that identifies the offset within the page, right? And the nice thing about doing this is we can essentially take the virtual page number, replace it with a physical page number and that's how we perform the translation, right? So for 4K pages, it turns out again that we can do some nice bit masking here, which again, you guys will do this for assignment three, right? You take the top 20 bits, that's your page number. You take the bottom 12 bits, that's your offset. You map the virtual page number to a physical page number. We'll start talking about how we do that today and data structures that the kernel uses. And if a virtual to physical page translation exists, we simply replace the virtual page number with the physical page number and we're done. Does this make sense? This is essentially segmentation and because we're using a power of two, right? We can do these bitmap operations. You could have a thousand byte pages. You could, and then you would be a mathematician instead of a computer scientist, right? Because then you would have to do real math, right? So let's say you had a thousand byte page just for fun. How would you, what would the page number be for a thousand byte page? How would you determine the page number? Anybody wanna guess? Tim, front row, Tim. So let's say I have a thousand byte pages and I have a virtual address of 10,000. What page number, oh yeah, someone said, oh, that's easy now, right? Gave you powers of 10, right? I have virtual address 10,000. My pages are 1,000 bytes. So what's my virtual page number for a virtual address of 10,000? 10, right? It's 10,000 divided by 1,000. What if I had the virtual address 20,072? What would my virtual page number be? Powers of 10 math. Looked at 20, right? I'm essentially taking the virtual address and I'm dividing by the page number and I'm taking that offset part that's within the page that's ignoring. Let's do some examples of these, right? Okay, here's my TLB, right? So my memory map, these are virtual to physical page number translations that the operating system has told the MMU about. So these are the virtual to physical page number translations and now we will use hex because it's kind of easy to do this, right? So if I'm using 4K pages, right? This is a hex address. Every nibble, which is actually what they're called in the hex address corresponds to how many bits? Who's good with their hex? Four bits. Four bits, four bits per nibble, right? So who can identify the virtual page number for this address? Bethany. 4K pages means the top 20 bits are the virtual page number. The bottom 12 bits are the offset. Four bits per nibble, 12 bits per offset. I just like saying nibble. I'm gonna keep saying that again.  Boom! Bottom 12 bits, right? Bottom 12 bits, four, eight, 12. This is my offset. Top other bits, right?  These are my page number, right? Does this make sense? Top 20 bits for a 4K page, virtual page number. Bottom 12 bits, offset, right? So you can literally do this. Just tear the hex address apart, okay? So where does this map to in the TOB? It is easy, right now. Yeah, so remember how my TOB works. I'm searching all these entries in parallel, right? This is content-adjustable memory. It's very efficient, right? And I get a hit, boom, right here, right? I look up and this is easy, right? We just didn't do this example yet, so that's okay, right? Now, how do I finish this translation, right? So I've identified the virtual page number for this physical page number. What physical address does this map to? Guru, who wants to tweak you? So, and what do I get? Oh, okay, so this is interesting. Do I add them together? What do you think, Paul? I replace the virtual page number with the physical page number, right? This is that the process of doing this is, we're doing it differently than we did segments. Yeah, great. So, okay, let's walk through this more slowly, right? The first thing I do when I translate a virtual page is I need to identify the page number, right? Because remember, the offset is the same in both the physical page and the virtual page, right? So here, right, this is my virtual page number, right? Hex 800, basically, right? This is my offset, right? The offset is gonna be the same in both the virtual and the physical page. So, I look this up in my TLB, I find that virtual page number 800 translates to physical page number 306, I replace the virtual page number with the physical page number, right? So I don't add the offset in, right? I'm actually doing that, right? I actually am adding the offset in, but I'm doing it using this trick by splitting the hex number into parts and replacing the top 20 bits, right? So now what I've done is I've taken my virtual page number and I've translated it to physical page number and this translation is complete, right? I wish I had an example here. You could do this with our old friend, the segment, right? So you could look this up in a segmentation table. All the segments would be the same size, right? You would remove the offset from the segment, right? Which is 346. You would translate the segment and then you could add the offset back in, right? It would work the same way, but because we're doing powers of two and because we're removing 12 bits, right? We can do this nicely here, right? Does this make sense? Let's walk through another example, right? How does, so how do I translate this address to a physical address? Let's step one. Yasha's. What's that? So what's, okay, what's my virtual page number? One zero, virtual page number 10, right? And what's my offset? Three 36. What physical page number does this virtual page number translate to? 50 and so what is the final address that this page translates to? Correct. Zero five zero three three six. How many people does this make sense to? Good. You guys are giving me these confused looks. It's like confused Wednesday or something and nobody told me, okay. So, okay, we talked about this, okay. So we talked about where entries in the TLB come from, right? So the operating system is gonna tell the MMU, here's an entry to put in your cache, right? This is very, very similar to segmentation and base and bounds, right? But what happens if a process tries to access an address that is not in the TLB, right? So a process tries to translate a virtual address and the TLB does not have a virtual physical page number mapping for that address. What happens, Alyssa? What can it do? What must it do? What, okay, so I have a better question. What can't it do? If there's no mapping, it can't do what? It can't access the address, right? It can't translate the address, right? And this is a load operation, right? Is there like a raise exception with this? Like is this Java, right? Could I throw an exception to the program? No, like this address either has to do what? It has to translate or it has to what? It can't complete, right? There's no way to complete this, right? So what does the TLB have to do? There were hands over here. Yeah, great. Ask the kernel. How do I, and how does the TL, how do you think the TLB asks, asks is the kernel for help is? Remo. It raises an exception, right? So this is our old friend, the exception, right? When there's a missing entry in the TLB, and this is now, I just wanna point this out, this is specific to a certain type of TLB architecture, but this is the one you guys are gonna be using for assignment three, so it's a good one to talk about. When there's a missing entry in the cache, the MMU calls for help, right? It says to the kernel, I don't know anything about this address, right? And then the kernel has to do, what does the kernel have to do at this point? The kernel has a couple of choices. What's one choice? Okay. I could terminate the process, right? What's a, what else might this result in? We might just call it a new, so. Okay, so if it's not, right, I'm gonna do this check, and if it's not a valid address, I'm gonna terminate the process. If it, let's say it is a valid address, then what do I need to do at that point? Frank. Yeah, I need to tell the MMU how to translate it, right? So I need to tell the MMU here's a page mapping, right, for this. So again, this is, these are, you can think about these as fixed size segments, right? Is the address in a segment? No, ask the kernel. Kernel says either there is no segment for that, as I never told the process it had permission to use it, I'm gonna kill that process, or it tells the MMU here's how to translate 4K worth of virtual addresses, right? So remember we talked about segments, we said that there was this trade-off. Each exception that the MMU caused allowed it to learn how to translate a certain number of addresses. So how many addresses does the MMU learn about at a time with pages? Tam. Four, or you were moving in the right direction with that answer and then there was a missing, Andrew? 4K, right? Four, I mean four would be kind of terrible, right? But at least this way, one exception and hopefully I've told the MMU, like I have, I've told the MMU how to translate 4,000 bytes, right? That's nice, right? Okay. So one of the nice things about pages is that as somebody was asking about before, frequently kernels still think about address spaces as being composed of segments, right? But all the MMU knows about are page translations, right? So what I can do is I can take a segment and I can basically break it into pages, okay? The segment, the ideas behind segmentation in terms of a segment having a logical purpose, right? And a segment having permissions that are associated with it, I can still just push those down to pages, right? So for example, if a segment is marked as read-write by the kernel, what does that mean about all the page translations that are inside that segment? Sarah, they're marked read-write, right? Page is just inherent, the permissions of the segment. But segments are kind of a nice way for the kernel to think about the virtual addresses, right? This is why again, when you guys mess up and see, I almost said a bad word. If you mess up and see, you still get a segmentation violation, right? What really happened is that there was a page translation missing for your process and it was killed by the kernel, right? But what the kernel tells you is you violated segmentation, right? So I can still do this, so I can still organize and protect regions of memory appropriately by just kind of inheriting these ideas from segments, right? This is even a better fit for address, better fit for address spaces, right? So I have less internal fragmentation than segmentation, because my allocation size is small, right? So I give the process permission to use 4K of memory and while that translation is loaded, it's allowed to use that 4K, right? But remember in a segment, I might have told the MMU, I might have to have a gigabyte of memory resident because I've told the MMU how to translate all those addresses, but the process might be only using a teeny, teeny piece of it, right? So this protrudes the internal fragmentation and also again, fixed allocation size means no external fragmentation, right? And this is fantastic, right? The kernel can never, the kernel will never not be able to allocate a page if it has memory available because if it has memory available, it has memory available in sizes of what? A page, right? Like I broke up memory into pages and I give out pages and all I get are allocations of one, two, three pages and again, the nice thing is, so what's also true? Let's say I have three pages, let's say I have four pages in the address space of a process that have been allocated, right? That are contiguous and the process asks for a fifth page. Do I have to allocate those five pages contiguously in physical memory or not? No, right? So I just find a page anywhere, right? If there's a page available, I can satisfy that request, right? So remember which segments those segments had to be contiguous. Every page is contiguous, but multiple page requests can go anywhere, right? I can map those page requests anywhere I want, okay? One of the problems with pages though is that I'm only telling the hardware how to translate 4,000 virtual addresses at a time, right? Whereas with segments, I could potentially tell it how to translate a million virtual addresses at a time, right? The other problem here, which we'll talk about on Friday is that this requires per page state within the operating system. And so there are all these tricks that the operating system is gonna start to play to try to make this state small, right? If the kernel had to save information about every page in the processes address space, that would be potentially bad, right? But what we want is we want a way to save only information about valid pages in the address space. All right, so the ideas with page state are, I have two goals, so on Friday what we're gonna talk about is how does, when the kernel gets involved, right? So we've talked essentially, we're pretty much done talking about hardware, we'll talk a little bit more about some hardware features on Friday that are different than what we've talked about, but we're pretty much done talking about the MMU. Now what we're gonna talk about is what does the kernel do when the MMU calls for help, right? And how does the kernel make this part of it as fast as possible, right? So when the MMU doesn't have a translation available, it's gonna ask the kernel to load one. The kernel has to keep all this state about the translations that are valid for each process, and so on Friday, we will talk about how, oh gosh, it's like 940. Well, maybe let's talk about it a little bit more. I'm sorry, my unit of time is off today, so I thought we were almost done. Somebody's peeking in the door. Okay, so let's talk a little bit about some of the state and then on Friday we can sort of wrap things up, right? I like this stuff, so maybe time's just going rapidly. All right, so again, now we're in the kernel, right? And then we have to think about what state, so when the MMU asks for help, right, what does the kernel have to be able to tell it? So first of all, what is the MMU going to tell the kernel, right? When the MMU raises an exception, what state does the kernel have to, what does the kernel have to know, Sam? Okay, it needs the virtual address or the virtual page number, potentially that it was trying to translate, and then what else might I need, Jeremy? Oh, yes, very good. Why do I need the process? Why do I need to know what process was running? Sumit. Well, but this is a separate thing, so I also need to know, so I need to know the virtual address that it was trying to translate. I need to know what happened, right? Was it a load, was it a store, was it a fetch, right? Was it trying to execute from this instruction? Why do I need to know the process? This is a great review question, because the virtual address has no meaning outside of a process context, right? It is, if I tell you translate virtual address 10,000, you cannot answer that question, right? Is it impossible? It means nothing, okay? Now, in actuality, the MMU doesn't have to tell the kernel what process is running normally. Why not? Well, okay, you can do a one-word substitution and that answer and it'll be correct. Who knows what process is running? Yeah? Yeah, the kernel scheduled the process, right? So the kernel knows, right? The kernel knows that, oh, yeah, okay, you know, like, yeah, that process is running, right? It's on the running queue I assigned it to the CPU. Now there's an exception that emerges from that CPU, so the kernel knows this, right? Typically, actually, the hardware doesn't really know about processes that much, right? The kernel has an ID and other things, so the idea is that the exception occurs in the context of some process executing, right? And the kernel needs to know that, okay? Okay, so I need to know what process was running. I need to know the virtual address that it tried to translate and I need to know something about what it was trying to do, right? What information do I need to tell the MMU when I'm finished? Well, I need to complete the translation, right? But specifically, now we're talking about pages, so we're using page language. What do I need to tell? What do I need to tell? Harish. I might need to tell it to permissions, right? Sometimes the hardware is gonna help me in enforcing read-only or read-write or read-write execute, right? But what's the most important thing?  Yeah, I was calling on yawn. Okay, right, so the location of physical memory, what do we call that? What's that? A physical page number. A physical page number, right? So the request comes in, it says, I don't know how to translate a load to virtual page number X, right? The process is aha, the kernel says aha, this was the process who was trying to translate this. And if that address is valid, right? That's the first thing I need to determine. If it's valid, I need to tell the MMU this virtual page number maps to this physical page number, right? So this is the transaction that has to go on, right? The data structure that's used to store a single entry, right, a single virtual physical page number mapping is usually referred to as a page table entry, right? Or we colloquially refer to them as PTEs, right? And we'll talk about why they're called page table entries, you know, in a slide or two, right? But a page table entry is designed to incorporate all of the information about this translation. And on modern systems, right, so first of all, there's gonna be a lot of these, right? The kernel will have a lot of these floating around, okay? How many page table entries, conservatively, will the kernel have to keep track of? By a five processes, each process, you know, just explain how to do the math one way. Okay, page table entries, sorry, not page tables, right? So one for, I definitely, it definitely scales with the number of processes, but what else does it scale with? It's positive. The amount of virtual, it turns out the amount of virtual memory that the process has allocated, right? So if I have five processes, if I have a system that is half gigabyte of physical memory, but I have five processes that are each allocated a gigabyte each of virtual memory, then I need to have page table entries for all of that virtual memory, right? Because it turns out that this page table entry is also gonna contain information about where the page is, even if the page contents are not in memory, right? So the page table entry is designed so the kernel can find out information about this specific page, right? It could be that that page table entry contains information about where it is in memory, but it also could be that it contains information about the fact that it might not be in memory at all, that its contents may have been moved, okay? So modern systems play a lot of games with these to make them small, right? So if you look at Linux and Windows, they have all of these bit masking things and bit packing stuff, right? This is not a structure that you just like allocate and allow it to sprawl over like 64 bytes, right? Like they try to jam this stuff together, right? So it's like, if the first bit is one, then I interpret the next eight bits as the whatever, and if the first bit is zero, then I interpret the next six bits as something, right? There's a lot of messy stuff that goes into trying to cram. It's much information. Ideally, you try to get these into like four bytes, right? Because usually like four bytes is an allocation size, but you try to make these as small as possible, okay? And there we go, right? So here's our goal, right? Jam everything into one 32-bit machine word, right? So some information about the location, right? That could be a virtual page number. If it's on disk, right? If I've moved the contents to disk, which we'll talk about next week, then I need to also, like wherever the page contents are, this location has to allow me to find, right? You know, the permissions, valid bits usually tell the kernel, is this page actually resident in memory or has it been moved? And then for, and we'll talk about this again more next week, for tracking the access to pages, I usually have some state in the page demo entry that helps me determine whether this page has been used, right, or how frequently it's been used. And this is something that's gonna concern us more next week, Jeremy, yeah. Yeah, yeah, that is, if you've set up a Linux system before and you've allocated space on your hard disk for swap, right? And swapping, which we will talk about next week, that's where pages go when they're not in memory, right? That's one of the places that the kernel will put them, right? It turns out that you should have, you have more than an academic interest in the format of these page table entries because you guessed it, for assignment three, you get to format these entries yourself, right? You will get to design your own page table entries and implement them and you will have to use them to figure out where things are, right? So again, usually the location, again, my location, if it's a 32-bit address with a 4K page is 20 bits, right? So frequently I can use four bits for read write, execute permissions, I might have one bit for valid and then one bit and look, you know, I still have a, I still have six bits left for fun, you know, to store, to store little messages I'm sending to the page, you know? I don't know, whatever. So yeah, there's tricks that people play to try to get all this stuff into a very, very small package. I think on modern systems I'm guessing that a lot of times now these are 64-bit structures, right? I should look on Linux and see how well they've done, right? I think at some point when machines started to have more memory, these structures kind of spilled over 32 bits, right? And then once you pass 32, might as well go to 64. All right, so let's talk about how this works, right? So the process is going to tell the MMU store to address 10,000, right? The MMU is gonna say, I have no idea where this address is, right? And it's gonna ask the kernel for help, right? What happens now? Just, this is a review from five minutes ago. Who, how does hardware get the kernel's attention? Throws an exception, yeah. We like to say raises, right? Once you say throws, then I feel like I'm worried you're talking about Java or something, right? Like it raises an exception, right? Of course you can be talking about Python if you say raises, right? But the point is that there's a electronic signal that is sent to the kernel by means of changing the logic value on a line that's connected to a port on the kernel, right? So yes, it raises an exception, right? So now we have this challenge, right? Because now, think about this, right? We want memory translation to be fast, okay? And the MMU is in general fast, but as soon as the MMU gets into the kernel and asks the kernel for help, what is already true about this translation? What do we want to happen? For most cases of virtual to physical translation, what do we want to happen? Wembley. Yeah, so once we get to the kernel, things are already going wrong, right? Not wrong in the global sense. This is a normal part of process operation, right? But wrong in the performance sense where we're already slow, right? Like this is already taking forever, right? And now if the kernel is like super disorganized and takes like all this time to find the translation, now it's just getting worse and worse, right? So if you think about it, you're already on a very, very sort of hot code path, right? And you're already taking like thousand times longer than you want to, right? So don't add, you know, misery to something that's already miserable, right? So the requirements for how we locate this information, there's two of them, right? And these two requirements are completely, this is another case where we have an interesting design trade-off that we're gonna talk more about. Because these two requirements are completely at odds with each other, right? The first requirement is speed, right? Again, we're on a hot path here. This happens a lot, right? And actually it usually happens a lot at a time where the process is already running slowly for some reason, right? So the process is already throwing these exceptions and I'm gonna slow it down more, right? So I wanna be able to do, I wanna be able to find my page table entry as fast as possible. The other thing is, if the data structures that the kernel uses to store these translations take up 90% of the memory on the system, then it doesn't matter how fast they are, right? Because now I'm forcing processes to use so little memory that they might not, I'm limiting the effectiveness of the system. Oh, yeah, great question. Where do these data structures live? They live in memory, right? And so if I see where you're going with this, can I have a page fault or can I have a case where the data structures that are necessary to translate a page address are themselves not resonant in memory? And the case is, yes, you can and we don't talk about that and you don't have to implement that for assignment three because it gets, you can if you want to, right? Mark Zuckerberg's partner in this class when he was a TA for the course declared that he was going to get this to work. For OS 161, which you can do, as far as I know, he never did, but it is possible we don't encourage people to try it, right? So it can be that and we'll talk next week about what happens when things get moved to disk, but on modern systems it can be the case that these data structures themselves can be moved to disk, right? So imagine that I'm in the process of trying to translate an address, right? And the kernel now uses a virtual address that itself needs to be translated, right? So now I have what's called a recursive exception, right? And the process of generating, of trying to address a page fault that was generated or sorry, a virtual memory exception that was generated by a process, the kernel itself can cause other exceptions with which it has to handle, right? So this can get very, very complicated. Okay, so it's a good question. You guys can start packing up. I don't wanna get into too much detail about this, but the reason why the kernel would use virtual addresses, especially for storing this type of information is the same reason that processes would, because if I have a virtual address, I can move it, right? If the kernel uses a physical address for its, so when you guys implement this for the VM portion of this class, we will tell you to put all of your kernel data structures for address translation in physical memory, right? And you will use addresses that are hardwired to physical memory. If you do that, there's no way that they can ever move, right? Once you use them that way, if you wanted to move them to disk, right, to free up memory for some other purpose, it's impossible to do it, right? So the reason why the kernel uses virtual addresses is the same reason that the process is, right? And that's why on your system, for example, there's a portion of the kernel address space that will translate, right? That requires translation by the MMU. All right, so on Friday we'll talk more about how to do this efficiently and fast, and if you want your midterm, come upstairs.


The instruction for the enlisted Information Dominance Qualification Program was updated on June 8th. This instruction contains major changes to the program, including clarifying which rates and commands are eligible to participate in the program. The Romania Aegis Assure Missile Defense System is one step closer to operational status. 200 sailors will man the site in the future. Robots from the United States claim second and third place finishes at the DARPA Robotics Challenge. The robots were given tasks such as driving alone, walking through rubble, using tools and climbing stairs to demonstrate their ability to function in situations too dangerous for humans. And finally, sailors worked to spread Navy awareness at the 2015 ESPN Summer X Games. The Navy hosted several events including a pull-up challenge, a chance to operate robots, and a tour of a special operations craft. For these stories and more, visit navy.mil.


So for this next hack we're gonna be showing you guys how to get inside Chick-fil-A on a Sunday when they're closed What? It's open. That's an employee Wait this looks and smells so good I'm gonna do this every time we go to Chick-fil-A. Hey guys welcome back to this week's YouTube video So this week we're gonna be testing out some TikTok food hack Kind of like the part 2 to the testing viral TikTok life hack. So forever who wanted a part 2. This is your part 2 I'm really excited to try out these hacks and also we're really really hungry as well So we're really excited to be trying some of these food hacks and if you're excited for this video ask Well, be sure to get this video a like or a hundred thousand likes once again, and yeah Let's get started with the video. Alright, so for this food hack It looks like it's a McDonald's food hack and let's watch the video right here Alright, so it looks like you need ice cream and two chocolate chip cookies and then you combine the chocolate chip cookies together Put the ice cream in the middle and you make like a little yes So it's like an ice cream cookie sandwich, huh? Yeah. Yeah, it looks good It's like the thing with McDonald's is like they have like really really good ice cream and really really good cookies So that's like a very good dessert combination that I've never tried out before so let's run a McDonald's and try it out Yeah, all right, so we just got back from McDonald's and lucky for us their ice cream machine was working So now we can finally put the ice cream cookie sandwich to the test two cookies per ice cream sandwiches What you're gonna need. All right. This is messy.  He just licked his food.  Here we go There we go. Look how good this looks three two one Hmm. Oh my gosh. It is so good. Why is it so good? I would rate this like a nine out of how you recommend this is one of the best desserts I've ever had I definitely recommend trying this out, but yeah, let's move on to the next life hack All right, so for this next food hack We are at Starbucks right now. We're gonna be testing out this drink hack Apparently it's like on their secret menu It's not even on the secret menu. You just have to request this drink. It's called a mermaid Frappuccino But here's the hack right here. We're gonna go check it out Okay, so it looks like you just add a whole minute vanilla bean extract two scoops of that Didn't get dragon fruit three pumps of cream syrup base lots of ice and then you blend it add matcha powder Dragon fruit dust and then there's the drink seems really really complicated But it actually looks really really cool like I've never seen a drink like this before like are they even gonna be able to make it You think I don't know it looks really really good. So we should just go inside and test it out Alan. So yeah, let's just do that All right, so we have a mermaid Frappuccino here But I'm actually really surprised if they've never heard of the renegade shake or that or the tiktok Frappuccino Actually renegade shake is just something to be made up. But anyways, here's the mermaid Frappuccino and it actually looks really really good Let's try it out. Here we go This is this might be my favorite drink it's like super complicated, but it's like so good I would definitely recommend it. All right So we're at Chick-fil-A now and this hack will actually let you get Chick-fil-A on a Sunday when they're supposedly closed And also we're gonna be trying out a food hack as well over here and yeah, let's just get started All right So for this food hack right here looks like he just got some chicken nuggets some fries ranch sauce and buffalo sauce And a salad container then he shakes it up and he's got a homemade buffalo style chicken nuggets and fries That actually looks really good. So yeah, let's go test it out on a Sunday. I'm excited to try it Yeah, I know and I'm also excited to see even get it on a Sunday. We used to work at Chick-fil-A. That's how we know this hack works Okay, so let's go test it out. All right, so this is the back entrance, but I actually seen an employee inside. So What All right, so we have all the ingredients right here we actually added a barbecue which is like optional But we just wanted barbecues just because we don't want it like too spicy also a huge bummer We thought today was Sunday turns out it wasn't The employee was pretty confused Anyways, we have our eight-piece chicken nuggets and the waffle fries and we already have it mixed in okay mix it in And now I'm putting in the barbecue. Oh this already looks really good. That is true All right, I just did a horrible job But now the only step left is mixing it put the lid on it and shake it up one two three Break it. Okay, here we go.  This is so good. I'm gonna do this every time I go to Chick-fil-A. I'm not even lying So you guys know we're not capping. I actually finished every single bite. There's nothing Alex. We're filming.  All right, so for this TikTok food hack We're gonna be making a watermelon smoothie first thing you need is obviously a watermelon The next thing you need is something sharp to cut like a decision point and the watermelon where you're gonna be and Then you're gonna place the it's easier to show you guys so step one We have to watermelon step two.  We got a different knife for it. It's the one that can cut deeper. Oh When you feel like you've made a hole big enough for the mixer to fit in you stick the mixer in so you want to blend it for about 30 seconds and completely liquefy the watermelon All right, so we just blended it for about 30 seconds take a look inside to see if there's any juice Oh my gosh, there's a lot of juice in it All right, so what you want to do next is you want to get like a little cup and cut it directly in the middle After you cut the cup in half you want to reshoot the opening so the cup can fit in there a little better Okay, so now the hole looks like it'll fit the cup. Let's see if it works and oh, there we go Oh, that's perfectly look at this. Wow.  All right moment of truth guys. Oh That's not bad here wait, you know what I have a better idea. Let's get rid of the pouring spout Let's just dump it in. It's definitely way better way. Oh My gosh, we don't need the cup. We don't need the cup. You can just drink it like this Wow, you know or better yet. You drink out of that. I'll drink out of it like this out go You want to taste test it taste test in three two one? mmm This is so good. I forgot how watermelon juice taste of like wait This is so actually so good and it's super healthy too. So you know like I mentioned I don't think you actually need a pouring spout if you don't mind pouring it in the cup Like I just did it seemed a lot easier and yeah, or you can just drink it out of it like this But I highly doubt this is a good idea because other people may not want to drink after you which I don't care So one more neat trick you can do with the watermelon You can actually turn a watermelon smoothie into a watermelon breezy by putting it in the freezer This is something cool and neat little trip that you can try with your friends and a watermelon and yeah I would recommend it and now let's move on to the next hack. Okay, so for the next food hack It's also a McDonald's food hack. Yeah, there's a video right here. Let's go watch it Okay, looks like you need a power drill and then you're gonna be putting snacks inside a McFlurry and you're gonna use the power drill to like blend it up and once it's done it should come out looking really really good That's what it looks like. Let's go test that one out as well. Alright, so we just got back from McDonald's We have our McFlurry and our snacks right here. We chose Kit Kats and Hershey's We also need a power drill for this obviously or a blender anything like that or a mixer Basically, we're gonna be putting these snacks inside the McFlurry and we're gonna be mixing it up And that way you have your own custom made McFlurry So let's put in the Kit Kats first two Kit Kats in each now onto the Hershey's this and this oh really? Yeah, alright So we're gonna mix it up Okay, so clearly I did something wrong. I didn't hold on to it I really want you to hold on to it this time. I'll hold on to it this time and you'll blend it Careful though, what if I cut your hand go? Oh my god. This looks good. This looks good. All right, so the taste test How is it? It's a lot better than it was earlier, but really because it's This food hack I would give about a seven out of ten. I'm gonna highly recommend this it's not I don't think it's too safe All right, so for this next food hack It's more of like a cooking pack. They're basically the videos right here. It's like an egg sandwich hat Yeah, so inside the video Okay, so you just put a little bit of butter inside the pan for the eggs and after mixing it get two pieces two slices of bread Get the bread in each side and then flip the egg over flip up the side Now flip up the other side in it and there's a sandwich right there. Wow. That looks yummy I'm gonna have to try that that actually looks so good. Let's go test it out first step We don't have butter, so we're gonna use cooking spray instead. It doesn't really matter so step one and Then what we want to do now is want to get every egg What we want to do now is we want to like whip it up start up to where it's really scrambly looking That looks good So you dip the first slice in like this and second one in like this let us sit for about 30 seconds All right, so right now we're just gonna flip it around over I didn't think you'd be able to pull it off all right So now you want to tuck the edges in that all right now then flip it around boom So good Looks so good in the video. This is how it came out like what is this do something wrong? Did we skip a step? I mean so turn out pretty good. I think I just used too many eggs I would recommend using two eggs and not three Alex will be doing his version of it and see if you can do a better job All right, I put two eggs in this time. Oh Wait it has to be even I know that this isn't good I'm gonna let it sit for a few seconds and then I'm gonna flip it around three two one. Oh Look at that. Oh, that looks good mine actually looks pretty good. All right taste test three two one This is a good breakfast. I need this for breakfast. I would actually eat this Yeah, I would I just have to get better at it and once I perfect it I would eat this every day I feel there's no flavor, but other than that like this is like a really quick way to make an egg sandwich So yeah, this was uh, this was not bad.  Okay? That doesn't look very good, but we're just trying this out for you guys. No, it might be good Yeah, let's just start crushing up the bag of Cheetos. Yeah. Oh, this is good Whoa, whoa, whoa, whoa, whoa. I don't have time for this. I thought he was gonna hit my head with it That's why I ducked. I was like whoa, whoa, whoa Whoa, hold on. I actually want to try another way because this is not really working out that well Okay, so I think whatever body slam you did to me worked a little too good because it ripped a hole in it Oh, hey, there we go. It's actually kind of crushed up nicely moment of truth You want to go first Alan into the honors you'll go and then I'll go and then we'll see how it tastes like I don't really like pickles. So okay, let's see. Oh, yeah. Oh, yeah. Oh, there we go.   All right How is it? I don't know. It's a lot better than just eating a pickle I'll say that doesn't taste that good, but it doesn't taste bad But I definitely want to eat it again on my own free will like all right my turn tastes like fried pickles This is not it. This is not the most sake of the video. I'm not gonna throw up, but this is bad I'm throwing this away. Oh my gosh. That was so bad. Please don't try that. Okay. Let's do the next half All right, so for this next hack, we're gonna need you to subscribe like the video It's a really good hack if you want to see more of our YouTube videos. Yeah, so by subscribing our videos will pop up more and hey like that's a good hack You want to see more of our videos? Yeah, please subscribe and like the video if you enjoyed it at all And thank you so much for watching once again, and we'll see you guys next time


From theCUBE Studios in Palo Alto in Boston, connecting with thought leaders all around the world, this is a CUBE Conversation. Hey, welcome back, you're right here, Jeff Frick here with theCUBE. We are still getting through COVID. It's a hot August day here at San Francisco Bay Area. I think it's 99, somebody said in the city, that's hot. But we're still getting through it. We're still reaching out to the community. We're still talking to leaders in all the areas that we cover. And one of the really interesting areas is natural language processing. And it's a small kind of subset where we'll get into a little bit more detail with a very specific place within the applied AI world. And one of my very good friends and CUBE alumni who's really an expert in this space, he's coming back for his second startup in this space. And we're joined by Ben. He's been chung, the co-founder of Ogmagod. Did I get that right, Ben? Ogmagod. That's correct. That's right. Great to see you again. Thank you for inviting me to the show. Well, I love it. You know, one of the topics that we've been covering a lot, Ben, is applied AI.  It's kind of this global big thing. And it kind of reminds me of kind of big data or cloud. In the generic, it's interesting, but it's really not that interesting because that's really not where it gets applied. Where I think what's much more interesting and why I wanted to have you back on is, no, where is it actually being applied in applications? Or where are we seeing it in solutions? And where is it actually changing people's lives, changing people's days, changing people's behavior? And you seem to have a propensity for this stuff. It was five years ago. I looked, July, five years ago we had you on and you had founded Genie, which was a natural processing company focused on scheduling. Successful exit sold that to Microsoft. I think they baked it into, who knows? They probably baked it all over the place. Left there, now you've done it again. So, before we get into it, what's so intriguing to you about natural language processing for all the different kind of opportunities that you might go after from an AI perspective? What is, what's special about this realm that keeps drawing you back? Yeah, sure. Yeah, I mean, to be honest, it was not anything pre-meditated. I kind of stumbled on it. Before this, I was more like an infrastructure guy, spent a number of years at VMware and had a blast there and learned a lot. Then I kind of just stumbled on it because when we started doing the startup, we didn't intend it to be a AI startup or anything like that. We just had a problem that my co-founder, Charles Lee, and I really wanted to solve, which is to help people solve people's scheduling problem. But very shortly after getting into and start looking at some of the use cases, we thought that the easiest way is to communicate with people like humans due to help them do the scheduling. And that's kind of how I stumbled on it. And it wasn't until that I stumbled on it that I realized that it has a lot of traction to me because throughout my whole life, I'm always very interested in the human emotions of it, how humans relate to each other. And that's always been a hidden side project thing. I do traveling to figure out those stuff and get a little bit of that. But once I started getting into this field, I realized that there's a lot about it, about humanity and how humans communicate, that it was kind of like a hidden interest for me that now suddenly coming out and it kind of just got me hooked. Right, that's awesome. So one of the things, and we'll just get into it, is people are a little bit familiar with natural language processing probably from Siri and from Google and from Alexa and increasingly some of these tools. But I think you kind of rapidly find out beyond what's the weather and play a song and tell me a joke that the functionality is relatively limited. So when people think about natural language and they have that as a reference point, how do you help them see that it's a lot more than asking Siri for the weather? Yeah, there are a lot of capability but also hopefully not offensive to some of the tech visionaries just as a guy who's dealing with it every day. There are also lots of limitation. It's not nearly to the degree of refinement like what might be in preach out there saying that the machines are gonna take over everything in one day. We have a lot of struggles, there's very basic stuff with machines. However, there has been definitely a lot of breakthrough in the last few years and that's why I'm dedicating my life and my time into this area because I think that it just, there's gonna be huge amount of innovation continuously going in this area. So that's at the high level but if you talk about in terms of artificial intelligence and in general, I think I have my own understanding. I'm more like an applied guy rather than academic. So what I'm gonna say might make some academic screen because I'm more like an everyday practical guy and try to conciliate these concepts myself. The way that I view is that artificial intelligence is really try to help mimic some human capabilities that originally thought that it's the domain of human. Only humans are able to do it but machines now try to demonstrate that machine can do it as though the humans could. So, and then usually people get that mixed up with machine learning. To me, it's actually quite different thing. Artificial intelligence is just like what I mentioned. Machine learning is just a technique or a science or way of applying like to leverage this capability, machine learning capability in solving these artificial intelligent problems to make it more achievable, to raise the bar on it. So I don't think we should use them into changeably artificial intelligence and machine learning because today machine learning is the big deal that are making the progress first. Tomorrow it might be something else to help improve the artificial intelligence. And in the past it was something else before machine learning. So it's a progression and machine learning is the very powerful and popular technique right now to being used. Now within artificial intelligence, I think you mentioned that there are various different domains and topics. There is like object recognition, use of image processing, there's speech detection, there's video and video, what I would call action or situation detection. And then there's natural language processing which is the domain that I mean that is really in that stage of where we seeing quite a bit of breakthrough but it's not quite there yet. Whereas versus speech detection and image processing actually has done a tremendous progress in the past. So and you could say that like the innovation there is not as obvious or as leapfrogging as the natural language processing. Right. So some of the other examples that we know about that are shared often from machine learning are say the visual thing. Can you identify a Chihuahua from the blueberry muffin which sounds kind of funny until you see the pictures they actually look very, very similar. And they always stated that Google in their Google photos has so many pictures, such a huge and diverse data set in which to train the machines to identify a Chihuahua versus a blueberry muffin or you take the case in Tesla if you've watched any of their autonomous vehicle stuff and their computer vision process and they have the fleet, you know, hundreds of thousands of cars that are recording across many, many cameras are reporting back every night with natural language processing you don't have that kind of a data set. So when you think about training the machine to the way that I speak which is different than the way you speak and the little nuances even if we're trying to say the same thing I would imagine that the variety in the data set is so much higher and the quantity of the data set is so much lower that that's got to be a kind of special machine learning challenge. Yes, it is. I think if people say that there is we are the cusp of being able to understand language in general, I don't believe that we are very far away from that. And even if when you narrow scope to say like focus on one single language like English even within that, we're still very far from it. So I think the reality at least for me speaking from ground level kind of person try to make use of these capabilities is that you really have to narrow it to a very narrow domain to focus on and bound it. And my previous startup is really that, you know that our assistant to help you schedule meetings that assistant doesn't understand anything else other than scheduling. We only able to train it to really focus on doing scheduling. If you try to ask it about a joke or ask anything else, it wouldn't be able to understand that. So I think the reality on the ground at least from what I see of a practical application and being successful at it you really need to have a very narrow domain in which you apply these capabilities. And then in terms of technology being used broadly in natural language processing, in my view, there are two parts of it. One is the input which people sometimes call natural language understanding. And then that part has actually very good progress. And then the other part is the natural language generation meaning that the machine knows how to compose sentences and generate it back to you. That is still very, very early days. So there is that break up. And then if you go further, I don't want to bore Jeff here with all these different nuances but when you look at natural language understanding there are a lot of areas like what we call topic extraction or entity extraction, event extraction. So to extract the right things and understand those things from the sentences there is sentimental analysis knowing that whether a sentence expressed is somebody's angry or some different kinds of emotions. There is summarization meaning that I can take sets of texts or paragraphs of texts and summarize is with fewer words for you. So, and then there is like dialogue management which manages the dialogue for the person. So there are like these various different fields within it. So the deeper you look, there is like more stuff within it and there's more challenges. So it's not like a blanket statement say like, hey, we can conquer on this. And if you're digging deep, there's some good progress in certain this area but some areas like, you know, it's really just getting started. Right. Well, we talked about in getting ready for this call and kind of reviewing some of the high level concepts of and you brought up, you know, what is the vocab? So first you have to just learn what is the vocabulary which a lot of people probably think it stops there but really then what is the meaning of the vocabulary but even more important is the intent, right? Which is all driven by the context. And so, you know, the complexity beyond vocabulary is super high and extremely nuanced. So how do you start to approach algorithmic, algorithmically, excuse me, to start to call out these things like intent or I mean people talk about sentiment all the time that's kind of an old marketing thing but when you're talking about specific detail to drive a conversation and you're also by the way converting it back and forth between voice and text to run the algorithms in a text-based system I assume inside the computer not a voice system. How do you start to identify and programmatically define intent and context? Yeah, just to share a little anecdote, like one of the most interesting part of since I started this journey six years ago and also interesting also very frustrating part is that especially when I was doing the scheduling system is that how sloppy people are with their communication and how little that they say they communicate to you and expect you to understand. And when we were doing the scheduling assistant we constantly challenged by somebody telling us certain things and we look at it as like well, what do they mean exactly? For example, like one of the simple thing that we use to talk a lot with new people coming on the team about is that when people say they want to schedule it next week they don't necessarily mean next week. What they mean is not this week. So it doesn't, if you like take it literally and you say, oh, sorry, Jeff, there is no time available next week and actually Jeff probably not even remember that he told you to schedule it next week. What he remembered was he told you not to schedule it this week. So when you come back to them and say, Jeff, you have nothing available this week next week and Jeff might say like, well, your assistant is kind of dumb. I like, why are you asking me this question? If there's nothing available next week just schedule it the week after next week. But the problem is that you literally said next week.  So they're like, this is a good example where they're like a lot of sloppiness and a lot of contextual things that we have to take into account when we communicate with humans or when we try to understand what they say. So yeah, it's exactly your point. It's not like mathematics, it's not simple logic. There are a lot of things to it. So the way that I look at it, there are really two parts of it. There's the science part and then there's the art part to it. The science part is like what people normally talk about and I mentioned earlier, you have to narrow your domain to a very narrow domain because you don't have the luxury of collecting infinite data set like Google does. You as a startup or any team within a corporation you cannot expect to have that kind of data set that Google or Microsoft or Facebook has. So without that huge data set, so you want to deliver something with a smaller data set so you have to narrow your domain. So that's one of the science part. The other part is I think people talk about all the time to be very disciplined about data collection and creating training data sets so that you have a very clean and good training data set. So these two are very important on the science part and that's expected. But I think a lot of people don't realize is what I would call the art part of it. It's really, there are two parts to that. One is exactly like what you said, Jeff, is to narrow your domain or make some assumption within the domain so that you can make some guesses about the context because the user is not giving you to you verbally or giving you to you in the text. A lot of us, we find out visually by looking at the person as we communicate with them or even harder, we have some kind of empathetic understanding or situation understanding, meaning that there is some knowledge that we know that Jeff is in this situation. Therefore I understand what he's saying right now means this or that Jeff is a tech guy like me. Therefore he's saying certain things. I have the empathetic understanding that he meant this as a tech guy. So that's a really art kind of part of it to capture or make some good guesses about the context. So that's one part. The other part is that you can only guess so much. So you have to really design the user experience. You have to be very careful how you design the user experience to hide what you don't know so that it's not frustrating to the user or to put car rails in place such that the user doesn't go out of bound and start going to the place where you are not trained for that you don't have to understand it. Right, right. Because it's so interesting because we talked about that before that so much of communication, it's not hard to know that communication is really hard, email's horrible. We have a hard time as humans unless we're looking at each other and pick up all these non-verbal cues that add additional context. And am I being heard? Am I being understood? Does this person seem to understand what I'm trying to say? Is it not getting in? I mean, there's so many of these kind of non-verbal cues as you've expressed that really support the communication of ideas beyond simply the words in which we speak. So, and then the other thing you got to worry about too is you said ultimately it's user experience. If the user experience sucks, for instance, if you're just super slow because you're not ready to make some guesses on context and it just takes for a long time. People are not going to use the thing. So I'm curious on the presentation of the results, right? Lots of different ways that that can happen.  But how do you do it in such a way that it's actually adding value to some specific task or job? And maybe this is a good segue to talk about what you're doing now at Ugman God. I'm sorry, I have to look again. I haven't memorized that one yet. So how do you, I think you, because what you're also doing if I recall is you're taking now an additional group of data and additional data sets in beyond simply this conversational flow.  As you said, you've got to do the analysis on it. But at the end of the day, it's really about effective presentation of that data in a way that people can do something with it. So tell us a little bit about what you're doing now beyond scheduling in the old days. Sure, yeah, I left Microsoft late last year and started a new startup. It's called Ugman God. And what we do is to help salespeople to be more effective, understand the customer better so that they have higher probability of winning the deal or to be able to shorten the sales cycle. And oftentimes a lot of the sales cycle got lengthened is because of the lack of understanding. And there's also as a, we focus on B2B sales. So for B2B salespeople, the world has really changed around a lot since internet came about. In the old days, it's really about tell it to explain what your product is and so that your customer understand your product. But the new days is really about not explaining your product because the customer can find out everything about your product by looking at your website or maybe your marketing people did such a good job that already communicated to the customer exactly what your product does. But you really to win out against other people, you really like almost like a consultant to go to your customer and say like, I have done your job almost like I've done your job before I know about your company and let me try to help you to fix this problem and our product fit in as part of that but our focus is let's fix this problem. So how would you be able to talk like that? Like you've done this job before like you worked at this company before how do you get at the level of information that you can present yourself that way to the customer and differentiate yourself against all the other people who are trying to get their attention all the other people sending them email every day automatically. How do you differentiate that? So we felt that the way that you do it is really have the depth of understanding with your customer that is unrivaled by anybody else. Now, sure you can do that you can Google your customer or they read all the news report know all the leadership follow them on social media. Right, they're supposed to be doing all the stuff right then they're supposed to be doing all this stuff. And with Google and the internet there's no excuse anymore. It's like how did you not do your homework? You used to have to get the yellow things. Yeah, why did you do your homework?  It's right there, go on Google. But the truth is that you have to be empathetic to a salesperson. A lot of people don't realize that for a salesperson every salesperson you might own 300 accounts in your territory. And a lot of times in terms of companies there might be thousands of companies in your territory. Are you going to spend seven hours follow all these 300 companies and read all the tweets check out the thousands of employees in each of these companies that are linking profiles look at their job listings look at all the news articles. It's impossible to do as a human, as a person. If you do that, you'll be sitting in your computer all day and you will get in the door to have a conversation with the customer. So that is the challenge. I felt like salespeople really put up the impossible task.  And if you screwed up because you didn't check then it's your fault. But then on the same time how can they check all 300 accounts and be on top of everything? So what we thought is that like, hey, we made a lot of progress on natural language processing and natural language understanding and salespeople what they look for it's a quite narrow domain. They're looking for some very specific things related to what they're selling and very specific projects, pain points, budgets related to what they're selling. So it's a very narrow domain. We felt like it's not super narrow. It's a little bit broader than I would say scheduling but it's still very narrow the kind of things that they're looking for. They're looking for those buying triggers. So they're looking for problem statements within the customers that relate to what they're selling. So we think that we can use, develop a bunch of machine learning models and use what's available in terms of the web, what's out there on the web the type of information out there. And to be able to say like salesperson you don't need to go and keep up and scan all the tweets and all the news and everything else for these 300 companies that you cover we'll scan all of them. We will put them into our machine learning pipeline and filter out all the junk because there are lots of junk out there. Like Nike does like, I don't know hundreds of news release probably per week and most of them are not relevant to you. It doesn't make sense for you to read all of those. So, but how about we read all of them? And we extract out, you know we, this is called topic extraction we extract out the topic that you're looking for and then we organize it and present to you. Not just we extracting out the topic once we get the topic how about we look up all the people that are related to the topic in the company for you so that you can call on them. So, you know what you want to talk to them about which is this topic or this big point and you know who to talk to these other people. So, that's net net what we do. That's, you know, that's really interesting. It's been a tagline around here for a long time, right? Separating the signal from the noise. And I think what you have identified, right is as you said, now we live in the age where all the information is out there. In fact, there's too much information. So, you know, you should be able to find what you're looking for to your point there's too much. So, how do you find the filter? How do you find the trusted kind of conduit for information so that you're not just simply overwhelmed that what you're talking about if I hear you right is you're actually querying publicly available data for particular types of, I imagine phrases, keywords, sentences, digital transformation initiative, blah, blah, blah. And then basically then coalescing the ecosystem around that particular data point. And then how do you then present that back to the salesperson who's trying to figure out what he's going to work on today? B2B salespeople, they start with an opportunity. So, the opportunity is actually a very concrete word at least in the tech, in the tech B2B sales. We know, we see the 60 stories in downtown San Francisco will validate that state. Yes, so it starts with the word opportunity. So, the output is a set of potential opportunities. So, it speaks to the salesperson's language and say, when you use us, we don't just say, hey, Jeff, there's this news article about trurios, that's interesting to you. Oh, there's a guy that, you know, a trurio that matches what you, the big kind of persona that you sell it to, we don't start with that. We start with Jeff, there are six opportunities for you. Opportunities for you at trurio, let's explain what those things are. And then let's explain to people behind these opportunities so that you can start qualifying them. So, get you started right away in your vocabulary in a package that you understand. So, I think that's what differentiates us. Right, right. And at some point in time, would you potentially just think in logically down the road, you know, have some type of Salesforce API, so it just pumps into whatever their existing system is, that they're working every day. And then it describes, based on the algorithm, why the system identified this opportunity, what the attributes are that flagged it, who are the right people, et cetera. Awesome, so what kind of data sources are you querying? You're designing all product wise. Yeah, as soon as Dave and John watch this, they're going to want to talk to you, I'm sure. But what type of data sets are you querying? There are lots of them. We learned most of it by through the process of working for salespeople, meaning that we work for salespeople, we maybe, you know, quote unquote, stand behind their back and see what they're searching.  They're looking at 10K, 10Qs. They dig up various, some people are very, very creative, digging up various parts of the web and find really good information. The challenge is that they can't do this to scale. They can't do it for 300 accounts, because for doing it for one account is very laborious. So there are various different places that we can find information. And it's not, in terms of the pattern that we're looking for, it's not just keywords, it's really concepts, we call it a topic. We're really looking for very specific topics that the salesperson looks for. And that's not just a word, because sometimes words is very misleading. For example, I tell you, one of the common word in tech is called Jenkins. Jenkins is a very popular technology, continuous delivery technology step. But Jenkins also happens to be a very common last name for people. Well, I'm always reminded of our Intel days with all the acronyms, but my favorite is ASP, because you could use ASP twice in the same sentence and mean two different things, right? Average selling price or application service provider back in the days before, we call them clouds. But the nuance is so tricky. So within kind of what you're doing then, and as you described, working within defined data sets and keeping the UX and user experience pretty dialed in and within the Rails, are there particular types of opportunities in terms of B2B types of opportunities that fit better, that have kind of a richer data set, a higher efficacy in the returns? What are you kind of seeing in terms of, great opportunities for you guys? We're still early, so I can't tell you that from a global view, because we are like less than one year old experience quite honestly. But so far, we are being led by the customer. So meaning that there is an interesting customer, they ask us to look for certain topics or certain things. And we always find it to my surprise, because, and that really is a, like I'm constantly surprised by how much is there out in the web, like what you were saying. Customer ask us to look for some things and I thought for sure, this thing, we couldn't do it, we can't find it. And we gave it a try and low and behold, there it is, it's out there. So to be honest, I can't tell you at this point because I have not run into any limits. But that is because we are still a very young startup and we are not like Google, we're not trying to be all encompassing, looking for everything and looking over everything. We're just looking over everything that a salesperson want, that's it. So I'm going to make you jump up a couple levels such that you've been thinking about this and working on this for a long time. There's a lot of conversation about machines taking everybody's jobs. Then there's the whole side tranche to that, which is no, it's all about helping people do better jobs and helping people do more higher value work and less drudgery. I mean, that sounds so consistent with what you're talking about. I wonder as somebody down in the weeds of artificial intelligence, if you can kind of tell us your vision of how this is going to unfold over the next several years, is it just going to be many, many, many little applications that slowly before we know it are going to have moved along many fronts very far? Or do you still see, it's such a fundamental human thing in terms of the communication that these machines will get better at learning but ultimately they can kind of fulfill this promise of taking care of the drudgery and freeing up the people to make what are actually much harder decisions from a computer's point of view than maybe the things that we think about that a three-year-old could ascertain with very little extra effort. Yeah, if you take a look at what we do and hopefully didn't sound like with underselling our startup but a lot of it really is we're taking away the time consumer and also grant work process of the data collection and cleaning up the data. The humans, the real human intelligence should be focused on data analysis, to be able to derive lots of insights out of the data. So, and to be able to formulate a strategy how to win the account, how to win the deal. That's what the human intelligence should be focused on. The other part like struggling with doing the Google search and you return 300 entries with 30 different pages and you have to click through each one and then give up after the first three. That kind of data collection, data hunting work, we are really should not, I don't think it's worthy quite honestly for a very educated person to deal with. And we can invest it back in helping the human to do what the humans are really good at is that how do I talk to Jeff and I'm going to get a deal out of Jeff? How can I help and do helping him solving his problem? How can I take the burden of solving the problem from Jeff's head and solve the problem for him?  Right, right. So, so what, so then that what I'm trying to say using ourselves as an example is that we're not taking over the job of a salesperson. There's no way that we can close the deal for you. But what we're doing is that we're empowering you so that you look like you're on top of 300 accounts and you talk to pick any of those accounts, you'll be able to talk to the people, your customer, that particular customer, like you know them inside out. Right. And without you being the super human to be able to do all this stuff. But as far as that customer is concerned it sounds like you were on top of all this stuff all day and that's all you do. You have no other customers. They're the only customer. In fact, you were on top of 300 customers. So that's kind of the value that we see to provide to the human is to allow you to scale by removing these grunt work that are preventing you from scaling or living up to your potential of how you wanted to present yourself, how you wanted to deliver yourself. Right. Well that's a great thing. There's no way that we can be smarter than the human in a way. I just don't see it. Not in my lifetime. I just love, you know, we've had a lot of conversations over the years and you talking about the difficulty in training, you know, the computers on some really nuanced kind of humane things versus the things that they're very, very good at. And you know, keeping the AI in the right guard is probably just as important as keeping, you know, the user interface in the right lane as well to make sure that it's a mutually beneficial exchange and one doesn't go off and completely miss the benefit to the other. Well, Ben, it's a great story, really exciting place to dedicate yourself and we are just digging, watching the story and we're going to enjoy watching this one unfold. So thanks for taking a few minutes and sharing your insight on natural language processing and the supplied machine learning techniques. Thank you, Jeff. It's always a pleasure. Yep. All right. He's Ben. I'm Jeff. You're watching theCUBE. Thanks for watching. We'll see you next time.