All right. Good morning, everyone. Hello. I guess we're all probably complaining about lab one Okay, so any questions from the previous lecture where we learned to create processes and had a lot of fun with it, right? Okay, nothing. All right, great because we're going to get into more of it today So again just in the recap just to make it crystal clear because the other section didn't like when what fork returned So how processes work is they're assigned a process ID or PID right people just call them PID because we just like to take shortcuts As soon as it gets created So that's assigned by the offering system and it doesn't change as long as that process is alive and remember a Process is just an execution of your program. So it's actually running and All the process ID is is just a number and it's unique for every active process So it won't change out from under you. So if you know, you create a process It will keep that process ID until it is terminated and we're going to see how to actually Remove it today because if you just call exit that doesn't Necessarily mean your process dies and we'll see what that means today But on most Linux systems, it's just a number, right? So it has some maximum for some reason the maximum is like 32,000 and then zero is reserved as like an invalid number or you know Yeah, just an invalid number to signal a return value And then eventually right you're gonna probably run out of numbers if you never reboot your computer So the links kernel will recycle a process ID, right? So if I'm process ID a thousand I die then it's I'm no longer running. So process ID 1000 is no longer in use. So if a new process starts a kernel could assign a process ID a thousand and and Remember the kind of unique thing about processes are just to give it a name is they have their own Address space and what an address space is it's just their view of memory, right? It's an independent view of memory one process doesn't affect another process So we saw as soon as we cloned it or serves we forked it We got two processes that looked exactly the same Down to their memory addresses and what their values were and then if we changed anything, right? The processes didn't affect another one So any questions on that which is kind of what we did yesterday, okay? So just to illustrate it super clear with that process diagram So this is how you would run some processes even on a single core So assume we only have a single CPU single core. So we can only one run one process at a time So Firefox something would launch Firefox for now. We don't care It would get assigned a process ID in this case one three three seven and It would be start in this created state where the kernel would have to load it in the memory and make You know set everything up initialize the program counter and all that stuff So then as soon as it's able to execute it goes into this waiting state Where we could actually run it if the kernel decided to schedule it to run, right? And we haven't talked about schedulers yet. We just talked about The actual mechanics of switching over a process So if I want to switch this process from waiting to running, what do I have to do? Anyone tell me the process I have to do or what it's called Yep Yeah, so I have to context switch right something else is probably running right now some other process maybe the kernel So I have to context switch and make this process running So at the very minimum I have to you know put whatever is currently running to Like retire whatever is currently running save its current registers at least its program counter Then store that somewhere that I'm going to reload the program counter for this Which would be initialized to whatever the entry point is and then I can go ahead and run it So say it's going it's running and then we create a tab Just the architecture of Firefox or Chrome whatever what web browser is they will Spawn a new process for every tab just to get that nice isolation. So if one tab dies the main browser doesn't die So as soon as this happens, right? It creates a tab if it creates a new process anyone want to tell me what system call it has to use to create a new process Fork right, so we're gonna fork So the kernel needs to handle the fork it would just it might just put this in a waiting state and Execute all the fork code because in this case we only have a single CPU So it has to steal the CPU away from this process put it back in the waiting queue and now the kernel itself is actually executing So it's part of that system call right the fork There's going to create a brand new process that is a copy of that but it would first be in this created state and it would copy that process control block copy all the memory and All of that fun stuff assign it a new process ID. So in this case one three three eight and Then it would set the parent process ID, you know equal to whoever called the fork in which case It was the original browser there, right one three three seven So it would be created in this case. It's a copy so as soon as the kernels done copying all the data in the process control block and possibly copying all the memory now It would go into this waiting and then at this point the kernel is free to schedule whatever one to execute at This point right you have no idea which one it will do. It's up to the scheduler to decide So say it goes back and execute some what main browser it would context switch back Go into this running code run it for a while And then it would context switch out and decide something else to run in this case, right? I only have two processes, so it would probably just have the tab running for a while And then after the tab runs for a while might you know It's time with the CPUs over goes back in this waiting queue and again the kernel gets to decide what to execute This case it could execute, you know the main process again And then go back right and each of these is a context switch So eventually the Firefox tab will be running at some point and then it will call exit and then go to this terminated state and It is still has a process control block empty So the kernel is still holding on to it It is active and the only way to remove it is we the parent process has to acknowledge its death And we'll see how to do that today. So only at this point It would go back to the main browser tab It would be running and it would have to acknowledge that that process died And it would at least read its exit code. So whatever got returned from main or whatever Was the argument to exit and we'll see that in some examples So in that case as soon as it acknowledges it The kernel can go ahead put this back in the waiting queue and then execute some kernel code where it would just Remove it and only now is that process dead So it actually takes a little bit from that exit call Another process has to acknowledge it before it's actually removed and actually dead So that at this point right the Firefox browser could go ahead and execute some more So we have some interesting questions now if Since that's the kind of Normal case where the child process will exit before the parent process and then the parent process gets to Just acknowledges it and then it can get removed But if there's a strict parent child relationship, right? Even in the last lecture we made sure our parent exited last by using sleep Well, the question now is what happens if the parent process exit first and no longer exists So in this case, right? If this process or sorry if the main Firefox process 1 3 3 7 up there died first then and Then got removed then whose parent would this Firefox tabs be if 1 3 3 7 doesn't exist anymore Anyone hazard a guess of what you might do if you have to maintain this parent child relationship and now the parents No longer there Yeah Yeah, so Yeah, so even based off this right it's a good analogy because we can already guess what happens when that parent That sounds awful when that parent dies, right when that parent gets removed Then it gets then that process without a parent anymore would get Reparented and it would go to the grandchild or the grandfather or might actually bubble up and we'll see What actually it gets reparented to because it's actually a little bit tricky But in general when you're writing this right whenever you have forking code You will create child children if you are forking and then the parent process is always responsible for the child Right going off this analogy The operating system whatever you terminate whenever you call exit on a process It will just set the exit status and then store it in the process control block and won't remove it So that process will actually still look like it's kind of active, but it'll get a special state that we'll see Right, so we can't remove the process control block It still has some information that the parent process needs to read before we the kernel can actually remove it and clean it up and The minimum acknowledgement is again just reading that exit status So whatever that return code was from the original parent or the original child process, which now is terminated so in this case there's going to be two situations for the case of That child so we kind of said the second one already that if the parent exits first It's called an orphan process because now its parent is no longer there And it has to get reparented somewhere and then someone else has to take responsibility for it The other scenario which is more likely that we didn't really cover before is this zombie process. So that's That's what it's called when the child has exited. So it's called exit and then no one has read that Read that exit code yet. So for all intents of purposes that Process can no longer execute anymore. It already called exit, but it's still kind of alive Which is why it's called a zombie. So it's dead, but still somehow alive again The kernel developers aren't Super They're not super imaginative. They just called it a zombie makes sense Yeah, so it's it's called exit. So someone needs to read its exit code So it still has an entry in the kernel and you could you know, you can read that process ID It's still active, but it can't run anymore. So it's called egg So it's called exit. It can't run anymore, but it's still around somewhere because we can't delete it yet Does that make sense? Okay So I just kind of said acknowledge without saying how to do it and I like giving code examples and making things very very clear so that by acknowledge people Mean the system call called weight and it has the following API and there's multiple different versions of it So if you read the man pages for weight, you'll see, you know weight wait Wait three wait for weight PID There's a lot of calls for weight that are all it evolved over time So the default weight system call has the following API takes one argument Which is the status so it is just an address to store the the weight status of a process It's not an actual integer So as part of the weight system call, which is why we need to understand pointers is You give it a pointer to an integer and it will write an integer to that pointer and then you can read it afterwards it's kind of a Roundabout way of getting multiple return values out a a function So it's normal return value that we're all used to is just a number That represents a process ID and if you look at the documentation It's a C wrapper for a system call So negative one again means there's an error and we set that error no global variable and Then there's kind of a special case here that we'll see is it will return zero if For non-blocking calls. So if you don't want if you just want to check if there is a child That you need to acknowledge you can just call it and have it return immediately to you and either it will accept a child In which case PID will be greater than zero But if PID is equal to zero when you make that call it means there's still a child It's still alive. You're still responsible for it, but it hasn't It's hasn't exited yet And then the normal case is it will return greater than zero for the child with the change and For all intents and purposes so far the change would be that it has exited So the weight status also contains a bunch of information Including the exit code. So if you just try and use it straight up as an integer It's not going to work because it only uses like the last eight bits for the exit code And then stuffs a but a bunch of other information at the front So to actually query it they give you a bunch of macros that you should use and we'll see that so you can use You know you can call man weight on your terminal to look at all the macros But we'll just see the main ones you need to use and then There's lots of different variations. We'll see later a weight PID to wait on a specific child process or Yeah, or you could also use weight PID as weight and we'll see how to do that as well So here is kind of an here's an example of weight So weight by itself is a blocking call So it will just sit there and wait and won't return until a child has died or it returns an error of some sort So let's read this program together. So we start at main The Something created this process. So a process will be created. That's running main and then the first thing we do is we fork So now we have two processes that are exactly the same except for the return value of fork So the child will get zero and the parent will get something greater than zero and that will be the process ID of the child So again, if it returns negative one, we're screwed We just return error. No, but in this case your computer is probably dying and then in the child, right? Which is process ID one or sorry zero It will go into that if branch and just sleep for two seconds sleep is just a way to yield CPU So it just says put this process to sleep for at least two seconds and then I'll execute again after that So child goes goes to sleep for two seconds and then it would go Outside of that whole branch go return zero and then that child would just die So the child all it does is wait around for two seconds and then exits with the code zero Then the parent It prints F that we're going to call wait It allocates some memory for an integer and it just does this on the stack, right? We remember our first year C courses So that just allocates, you know four bytes on the stack And then we can just pass its address to weight and then after and then it will write those some bytes to that address So after weight returns We can check the status since that's the only thing we said we I just Deleted the error checking right we would have to check if weight PID equals negative one and then there's an error I cut that out of this slide But then we would check If the process has exited so you we would use this macro so Wait if exited W status so that will check whatever bit they set for that And then if it's exit if it exited will say that hey wait actually return for an exited process and this was the process ID so which is the return of weight So we'll say what process ID is now dead and then we can just print its status So that's its exit code, which would be that integer got returned, which is really actually just a byte So in this case it should if the child just sleeps for two seconds and then return zero it should just We should get a zero red So let's go ahead and look at that So if we execute that we call weight and then this is the main thread and then after two seconds We see hey wait return for an exit process tells us the process ID of the child That is now dead and its status So any questions about that? Yeah, so it'll call those you know print F calling weight Which would just print that in the terminal and then it calls weight and the kernel will just put this process Just like just block this process until that child is actually done So if you look at weight weight only waits for your child Yeah, yeah, so a process can only acknowledge its direct children is how it works Yeah, yeah, so that's a good question. How does weight work with multiple children? I'll try and come up with the example at the end But if you read the documentation of weight it says it will return on the first Process that dies. So if you have multiple children here Say yeah, say we have two children here and we call weight It will just return on whatever the first ones that dies and the order is not specified It's just whatever dies first. So if you want to acknowledge both of them, you have to call weight twice Yeah, so there's a way to see if all the children are dead So we'll see a bit later in the other example. You can wait on a process ID Oh, no, that's next lecture but you can wait on a process ID or you can query the operating system and say hey Do I have any children and it will return a value for you? but in general right if you are the process unless you adopt some children which is Something that sounds like it could happen, but actually to adopt children. It's kind of an opt-in system So you only have to control the children you create. So you should know how many processes you create yeah So the status of zero is because in the child right we go to sleep for two seconds and Then we return from main with zero. So it's whatever that is. So if I want to see it change I could return. I don't know three here, right? So if I go ahead and execute that After I call weight the child sleeps for two seconds dies returns three and then I can read three Yeah This there's two processes right we forked at the first Yeah, so the child would just you know go into the if statement because its process ID is equal to zero It wouldn't go into the else so it's another process that executes the else which is the parent Which is the thing that has to call weight Yeah So you need to acknowledge it because something has to read its exit code, right? If if that child just slept for two seconds and then you know said it takes it code is three Which means something catastrophic probably happened to it and then it just gets removed completely from the kernel No one can ever see that it had a problem, right? someone needs to at least Acknowledge it and read its error code and that's the way the kernel checks that at least something has at least attempted to check its error Yep Yeah like here From the child So from the child if you want me to print here, so in the else or here Okay, right above the return zero print. Yes Yeah, so If we do that that'll get printed well here I can disambiguate when what they are Because that's returned from for right okay, so they'll get executed twice one in the child run one in the parent But because we're waiting the parents waiting for the child to die That should always get printed first because The parents still waiting for the child to die right sounds terrible God So don't worry the whole waiting to die it will get worse in the next lecture So if we call weight Oh Right I ever yeah, sorry I put a return there Yeah, so if I do that then I will always get here with the child first because The parent has to call weight, so it won't go past this weight until the child died So it will always print here zero before which is from the child and that's actually a Good segue into later lectures where we have to ensure order between things so that's one way you can ensure order between things Yep Oh because I just deleted the I had to return three here. So yeah No, the the status code for a process is whatever Got Whatever number you gave to exit so there's two ways to give a number to exit You can just call exit with a number or you can return the main with a number and then see we'll put that into exit 40 yeah, so if I return here in the child That's a return from main in the child So the child will exit with the error code 3 and that's why I didn't see that print statement at the end because the child's dead by that time Yeah, sorry if you call weight If you call weight in the child Yeah Yeah, so if you just call weight the parent process will be blocked. So it's waiting for something to die So it can't resume because it's just stuck at that weight call. So it's just waiting for something to die again sounds terrible Okay, well so that is what you're supposed to do. Yep So it's one. Yeah, it's pretty much the only way to establish some well No, there's lots of ways to establish order between parent and child. This is one of it So you can just wait until it dies so it The parent will just wait until the child so it can sleep as long as it wants right Okay, so let's get into zombies now and prove that they're actually a thing So a zombie process is just waiting for its parent to read its exit status again It can't be removed by the kernel because if it has an error something something at least needs to acknowledge it So it's been terminated so it can no longer run But it hasn't been acknowledged so something hasn't called weight on it so Again process may have an error in it or something where it never reads the child's exit status and that's you know That's up to that user program But ideally you would read the exit status of any processes and check that they're by convention non-zero Which means they have an error so another thing that we'll see next lecture is back into the land of interrupts So an operating system can actually interrupt a parent or process and kind of prod it to acknowledge a child So you essentially get a interrupt That we'll see on Unix systems are called signals It'll be like hey a child died So you don't have to just wait in a blocking call and we'll see some alternatives to that in the next one But again, it's just a suggestion. You're free to ignore it. And that's a form of IPC so Interrupts, you know our way to communicate between two processes In which in this case if the child dies the kernel can you know prompt your process to try and acknowledge it But you can also send signals to processes yourself to try and get them to stop executing which you've probably done before Or you know the kernel might also just send a signal to kill something if you sag fault it But the operating system again has to keep track of that zombie process until it's been acknowledged and then If the parent ignores it and it dies Right, the zombie process needs to wait to be reparented or adopted And then you kind of have this weird situation where that process is a zombie orphan Whoo that terminology But let's just check the zombie example real quick So don't worry about that print state code All right, so in this program again, we're going to this some Process is going to get created that starts executing main. Then we're going to fork two processes We don't know what order they'll execute in at this point But there's now two of them that are clones of each other except for the value of PID of return PID So again, let's ignore errors and then in this case We're going to the child's just going to sleep again for two seconds and then would hit return zero and then exit So in the parent I'm just going to sleep for one second and again the child is alive for two seconds so I only wait for a second and then I read the Processes state and you can see the code if you want this but this basically just checks that proc file system So it knows the process ID of the child from the fork So you can go ahead and read its status and do whatever you want So that's what print state does. It's just going to open, you know proc whatever the number of the child is and then Kind of parse the status file there So I have it to just print the state and in this case it should hopefully print the state If there's a problem with the print state function it returns error number from that Then after two seconds I print the state again and in this case It should be a zombie because it has died now because main takes a total of three seconds, right after one second It'll read it. It should be alive. Anyone want a hazard to guess as to what state the child will be in after one second So I Guess the lengths term all these kind of dumb would it be running? Would it be a zombie Would it be sleeping? You got like okay Well, it should be sleeping because we told the child to sleep for two seconds And then we woke up after one second and we're seeing what state it's in Whoops So if we do that We check first state it's in is sleeping and then we wait another two seconds check it state again and See that it is now a zombie because we haven't called weight on it yet, right? We're still right at this point. We're still terrible parents All right, any questions about zombie? Just this is basically just showing it does exist and it is a thing All right, so that is a zombie I press the wrong button All right So an orphan process if we go the other way an orphan process needs a new parent So it's lost its parent process. It is no more and the child Process still needs something to acknowledge, you know, it's exit code Well, the operating system or in this case the kernel specifically because it would be executing kernel mode Need to reparent the child and the default choice for most UNIX based operating systems is it will just Reparent everything to a knit So now we know the second job of a knit so The unit process is now Responsible for acknowledging all of the orphan so it is kind of by default the you can think of it as the orphanage So it has two jobs, right? so You can actually write your own in it now if you wanted so and it just has two jobs Remember when the kernel boots up it just creates one process and then that process is responsible for creating every other process And it would just do it through fork and exacting a bunch of stuff So if you want to write your own a knit you have to create every other process that's on your machine You know as soon as you create one generally they'll just kind of spiral out of control And then you also have to kind of acknowledge all the Orphan children that come to you that may or may not be orphaned zombies at that point, right? and then so if you look at the code for a very very simple a knit Because Because it you won't know the process ID of all the child since they get reparented to you So that's why wait the default wait just waits for any children that comes So if you're a knit all you do is you constantly just call wait wait wait wait wait wait wait wait wait wait wait So you can accept literally any children that any child that comes to you All right, so let's look at the orphan example and See where it actually gets reparented to so in our orphan example, we'll do the same thing we always do We will immediately fork. So now there's two processes We will ignore error codes because that's what we like doing Then in the child process, which is still alive. We'll just call this get P P ID So get the parents process ID So the child just immediately prints off what its parents process ID is which would be the original process Sleeps for two seconds and then it prints the its parents process ID after waking up for the sleep and then it would return zero and then in the parent it just sleeps for one second and then it Then it returns zero and now it's dead and thankfully Anyone hazarded a guess of what would read the original processes error code Or what called wait on the original process? Yeah Yeah, and what would the parent of that process be? Yeah, so like likely it's the shell so Generally shells are written correctly and they will you know wait and that's another thing the shell will do So that's how it you know prints off the exit number because it waits on whatever process it spawns But if we go ahead and run this it will look weird So the child's parent ID is you know 3006 and then we wake up again after our parent is dead and Deleted as well because if it was still a zombie and the shell for some reason didn't read its exit code That it its parent would be a zombie So it wouldn't get reparented because it's technically kind of still alive But in this case our shell works as we hope and then we see that its process ID after sleeping is 333 and By default I said it should get reparented to a knit which is you know Process one which is the great great great great grandparent of every single process on your machine So let's figure out what process you know 333 is so we can go ahead and look at our proc file system Read the status and let's just print off its name So if we print off its name, it's something called system D Which is what I told you is generally on Linux systems at least process ID one so this is system ID for your actual specific user and It's kind of like an opt-in so you could actually create a process that will Kind of opt into the kernel to re-adopt everything so but instead of Instead of being called like an orphanage or something that sounds I guess more friendly the name of it of something that accepts children is Called a sub reaper because it just kills everything. I didn't name it. Don't look at me So yeah, instead of trying to go the nice one, you know the kernel developers decide to call it a sub reaper and Process ID one is by definition the reaper. So it kills everything So nice and friendly right especially on what a Monday morning All right, any questions about weight? Yep. Yeah, so the shell would have a parent I think I showed it the last lecture, but the whole The whole thing on Linux is generally it would look like a knit and then a knit something and it would Probably spawn like that user system ID and then the process under that would probably be my gooey interface here Like this thing in this case it would be called gnomeshell and then in this gnomeshell would launch Visual studio code so the parent of visual studio code would be gnomeshell and then Visual studio code has a built-in terminal so the terminal would be a child of visual studio code and then The shell runs within the terminal so it would be a parent of that and then your process would be under that Yep, no There's no resurrection in Unix land it is dead dead Yeah, so it'll get reparented to the next sub reaper if you go up So if you just follow the chain of parents right because it's unique it'll go up until it hits a reaper And in this case just on this It's system D if you go ahead and you have a Mac and you download You know the examples and you execute it because they also work on Mac You'll see that it gets reparented to process ID one which is more what you expect So if I run this not on Linux if I run it like directly on my Mac it gets reparented to one All right, any other questions?  Well, I guess let's see what happens. So What got reparented to was process what? 333 we'll see in the next lecture how actually send signals, but this is where the terminology gets even worse so to send a signal to a process you would think that Something nice called like send signal or something anyone while hazard a guess based off like our whole rampage so far what it's probably named Yeah Death note really close It's called kill So this is also why if you say fault you'll see like killed thing because All right first off. I saw tons of some piazza. Can anyone actually tell me why a sag fault happens? Yeah Yeah, so if you try and access memory that you don't have access to right so it's unallocated stuff You shouldn't be able to touch that's what a sag fault is and a lot of you are getting safe thoughts in lab one so Most of them that I've seen are like so You're opening a big file right and then you think it's Doing like C string stuff on it, but if you just read the contents of a file What's unique about it a C string? Was it and what's a C string end with a? Zero right if you just read the contents of a file Does it end in a zero? No So if you treat that whole file as a C string Which just what anything you do to a C string right? It's just going to kind of iterate over it until it hits a zero So what do you think happens if it iterates through it until it hits a zero, but it doesn't have a zero in it Yeah, so it's eventually just going to fall off the edge of it right and then it's going to access memory You're not allowed to access and then it's going to say fault and That seems to be like 99% of posts on piazza So all right, so let's get back until this fun silly thing. So that is just an aside So and also the side effect of when you say fault the kernel will send a signal to that process to kill it And then it will get removed So I can kill in this case killed-9 means was it?   So It means the kernel just kills it. You don't have a choice whatsoever So we killed-9 this and if I do that yeah, that virtual machine's dead So I killed the parent of all my user stuff, right?  All right. So someone wants to get out of here early So, yeah, if we kill process ID one So either One or two things could happen Either if I kill process ID one, right? That's the root of every process That's the only thing the kernel cares about if process ID one is gone your whole thing is gone, right? Your user space is dead.  You'll work brick your whole computer. I'm actually not sure what it does Do I want to save everything now? Let's press enter Smart yeah All right. So that's a good comment. So Operation not permitted well, it might be permitted because root owns that so yeah, let's suit oh Colonel smarter than I guess the Colonel smarter than us. So you're not allowed to kill one. I'm pretty sure Yeah, it doesn't even give you an error message.  I can try and kill one as much as I want So obviously there's a special case in the colonel because otherwise it should just die, right? So Colonel protects you from yourself a little bit, you know, which is also why I'm still scared of doing this But if anyone's tricked you into doing that and deleting everything The you the craters of RM are smart and they detect this case and they won't let you do this anymore I'm not going to press enter just in case because I want to set that up again But that should be prevented yeah to one Okay, so a sig term is less severe than that so sig terms can be ignored It's just a suggestion That bash nine is a real one so it's called kill so it's like kill kill double kill like uber dead You're dead Okay Well, I guess let's wrap up then So there's a sletcher right you're responsible for managing processes if you are creating processes We should all be responsible adults here the offering maintains that strict child or parent child relationship And you should be able to identify and also kind of prevent the following Zombie processes, which means the child has exited first and you need to acknowledge it by calling wait and then You might also you don't want to create orphan processes, but if you did they will get reparented Likely to a knit for the purposes of this course will just say they get reparented to process ID one because that just makes things easier and Then process ID one would acknowledge them and then clean it up if it didn't write you up If process ID one didn't clean up all of the children The zombies would just start piling up right and if there's a maximum number of process IDs, which is like 32,000 As soon as you hit 32,000 zombies that aren't cleaned up. You can't launch any new processes Yeah, you can so in some situations you might want to do that, but you should Know what you're doing at that point so Yeah, so like Let's go back a lecture real quick So if we did this multiple fork example now that we can kind of understand, right? multiple fork was it Alright because I killed everything so in the multiple fork example where we created new processes and we didn't we never acknowledge them But all the new processes outlasted us anyways So all the main process did if we remember it just launched in this case for new processes So it just all we had to do is launch for processes and then all the children did something that took way longer So in this case, we know we're gonna die first We don't really need to acknowledge all the children because we know we're gonna die first the shell will acknowledge us So also to understand this too then since we die While all the children are running they're going to get reparented to something so then when they exit something acknowledges them Yeah Yeah Yeah, so the simplest in it will just have an infant loop that just calls wait Yeah, so most of the time it'll sit there just waiting for children Because most of the time it just launches some processes and then they do some more fancy work All right. Cool. So you should be able to identify those two things and hopefully play around with processes now So just remember pulling for you. We're all in this together


Okay, so welcome back. I hope everybody had a nice lunch break and Now I'm very happy to introduce Jungle call developer, but more importantly for this talk and employee of elastic search the company and on the ground Hello So I'd like to tell you a story It's a story about how we developed five clients for us to search in five different languages Without losing our minds in the process much And so as any good story it starts a long time ago in a galaxy No, no, no it starts actually when we looked at the current landscape of the clients for elastic search and There were some things that we liked and what we've seen are good and some things not so much For example in the in the python landscape. There are many clients, but none of them actually implemented the entire set of APIs none of them did Everything that we would like to see in a client and none of them did it On a scale that we would we would be comfortable with as a result users had inconsistent experience with elastic search itself and naturally they they blamed elastic search because of the way they Interfered interfaced with it was not ideal So we decided to create our own clients sort of to control The the last mile how people talk to elastic search so we can make sure that their experience is good and consistent So we started with the with the design obviously and We sit down and and set all the things that we want our clients to be and for that We need to start with elastic elastic Sorry elastic search itself Elastic search is distributed that brings a lot of problems with it and a lot of opportunities It talks via the rest API over HTTP Which is both good and bad because it's good that it can be deceptively easy to create your own clients Then the number of clients just in Python or just in Ruby was staggering just because everyone thought that oh, it's just HTTP Right, I can just do an HTTP request and everything will be good, but there are a lot of corner cases there that they didn't count with There's also a lot of diverse Deployments of elastic search some people just deploy The cluster in their own networks and talk to it directly others would use load balancers Some people would use alternate transport like thrift to gain some to gain some speed or would use a set of client nodes and a Distributed multi rack setup or something like that And also just a set of endpoints that elastic search has is quite staggering It's almost 100 API endpoints with almost 700 parameters that the clients only to support and document but More than that we wanted the clients to be true to their language That's why we only developed the four or five in the beginning because those were the people that we had We we had a we had a Python engineer yours yours truly here We had a we had an excellent Ruby developer a pearl developer. We even managed to find a really great php guy Believe it or not So those were the clients that we started with because we felt confident that we can actually make it feel Like a pythonic library not like a library written by it by a Java guy in his pair of time in Python and We wanted the client to be for everyone We wanted people not to have any excuse to use it So that's my first lesson that we learned No opinions no decisions in order to make sure that everyone would use the client We had to abstain from any making any observations any decisions because whenever you have an opinion There is someone out there who would disagree with you So the only way how to make sure that that won't happen is not to have any opinions So we decided the client should be low-level just essentially a one-to-one mapping to the rest layer And they should be extensible We should design them in such a way that where you don't like some aspect of it You should be able to replace it or just hook into it and change it So we came up with this This is a foreign for an HTTP client This is a kind of complicated diagram that specifies how the client works You have you have the client itself that has a transport class Which has a serializer to serialize and deserialize data as they go over the wire Then you have a connection pool that actually stores a list of connections Connection pool in this case is a misnomer because it actually doesn't pool collections It just holds a Collection of connections to individual nodes in the cluster You can see why we would have the naming problem there and then we have then we have connection by default we use your lip 3 because that Ended up to be the best one for for Python and we also have a connection selector So when you connect to multiple nodes, they control the strategy on how do you do load balancing? Do you use random or do you use round robin by default? We do round robin over randomized lists of nodes and The goal why we did it this way is so that we are able to give you the option of overwrite any simple component in here just by subclassing the default implementation and filling in all the blanks So some examples If you want to create your own selector you just create the class and you pass it in everything is essentially Using dependency injection so you can just pass it in as a constructor parameter and we will use that instead So you see three examples here. The first one is not really Not really injecting your own code it just setting up the options. So the first one instructs the client to Talk to the talk to the cluster and get the current list of nodes on startup Then whenever a node fails and then also every every 60 seconds. This is excellent for a long-running process Let's say a web server So that even when you keep changing your your elastic search cluster you keep adding nodes and nodes keep dropping out You still talk to all the nodes that are available The second one is where you want to control the load balancing For example imagine a scenario when you have two racks and you want to buy default only talk to the elastic search nodes in The same rack as the application server and only fall back to the nodes in the other rack if none of those are available You can do that. You can just write a simple class that will that will do this So that's that's what we Mean when we say that we are modular and extensible The last example is just using a thrift connection, which we actually provide as a as an optional plug-in and Using a different serializer in this case Yama because why not some people like Yama better than Jason for some reason So that was that was sort of the first lesson no opinion. So people have no excuses not to use it The second lesson was to prototype everything Because you just don't come up with something like this without some preparation without some prototyping and More importantly, you don't come up with it for a single language and have it be applicable for all the others It's very difficult to find a pattern that would work for both Python people and Ruby people for example That's why we created a spike of a Prototype implementation in both Python and Ruby sort of to make sure that The design will work that the design will hold for both of these languages and also So that we have a reference That we can talk about that we can we can have the same terminology and then when we talk about connection pool We know what we mean even though it means different things in different languages and even how we use it It's not exactly correct, but we had a code that actually showed what it does So we could have at that point a clear conversation even with even with a php and pearl people Even with the JavaScript people that came on and even later with with net people who were developing the new client now so Prototype everything not just to see if your if your design works But also that you have something to to talk about that you are absolutely certain that you're on the same page Because you should never trust humans and just their understanding if you can do more so That's the that's the next lesson The next lesson is don't send a man to do a machine's job Humans are amazing. They are amazing and a lot of things consistency not one of them Consistency and repetitive tasks you really want to have something that doesn't get tired That doesn't get frustrated that don't doesn't mind doing the same thing over and over again To me that sounds like a computer So this lesson states that you should automate as much as possible and This why I'm talking about this is I Already mentioned we have almost a hundred API endpoints With the 700 parameters That's very difficult to track That's a lot of work a lot of boring tedious work that you don't really want to be doing you don't hire a decent Python being and Forced them to maintain a list of 100 APIs and 700 parameters if there is any other way So what other way is there? So first we thought that we could do a reference implementation We can just arbitrarily choose one of the clients and decide this is how it should this is The reference implementation for our API's This is the authoritative collection of all the API's all the parameters their possible values and descriptions But that doesn't really scale that well first of all, we only have one Person per language. We only have one Python developer. We only have one pearl guy. We only have one PHP guy So What if what if he leaves what if he's on vacation and there is a change that needs to be made like and also How do you how does that person make sure that's everything is everything synchronous? We found out even with the spike implementation of the of the transport layer that Maintaining it when we add more features and we need to add it to both Python and Ruby even though it was just two of us and Coincidentally the two of us that lived in the same city, which is not true for any other two people on the project It was very difficult. It was difficult to keep in sync So we discarded reference implementation as an option Next we looked at documentation Because obviously elastic search has documentation and all the API's are documented All the parameters are documented as well, but Again, they're documented for humans. It's a documentation that's intended for for the developers to read it and To understand it to make sense of it So again, it would require a tremendous manual labor just to make sure That everything that we need is there Someone would have to read actually all the documentation collect all the All the stuff and not just one person, but each and every author of the client would have to do that That's a very tedious job a job that I I haven't signed up for and I I Doubt we would ever find a person who would sign up for it for a job like that So what are their options? We're there We found the progress from from reference implementation to documentation had some Had some promise, but it wasn't there yet So we decided to take it one step further To actually extract all the information that's already in the documentation. That's already in the code and presented in a in a structured format So we chose a format that's human readable and machine parsable and we had elastic search. We really love JSON So we just decided that we should document everything in JSON and create a spec of former specification For our API's This is the one case where I was super happy that our six searches written in Java as a statically gen type language Because it provides you with a bunch of tools So we were actually able to write a tool that would just parse the source code for all the API's and extract 90 80 percent of all the API's and its parameters in in an automated fashion and we then just had to go once over it and We could actually share in this effort all of the client people and just fill in the caps Fill in the documentation for each of the parameter Fill in the option and the type whether this Option is required or not whether it's a list on or a single value a boolean or an integer so that made the effort so much easier and Going forward also so much easier to define So what did we choose to to capture in this in this document? first of all the The URL path all the different variants of URL path if you if it was dynamic Which most of the URLs in elastic search are it can optionally Include an index name or a list of indices on on which to perform the the action it can Include other dynamic dynamic parts. So we have to document those including all the different options how the URL can look as part of it also We had to do the HTTP HTTP methods. So is it a get or post? We decided to do very little than that Just to list all the parameters list all the ways how to combine them into URL We didn't actually capture all the dependencies That this parameter is only valid if this parameter is set to blah We just found out that we don't really want to have this information this validation in the client And we'll instead decide on our users to use it directly So that way we would have less overhead with maintaining it and it would the code would be the code would be much simpler So how does it look? This is this is an example For for the suggest API. This is just a fraction of it and you can see that we have a link to the documentation We have all the possible HTTP methods. So in this case post or get We have the all the different forms of the URL paths with the description for each for each part So here there is only one dynamic part, which is the in optional index and We have description of all the parameters We also have a description of the body What it contains an information whether it's required or not and This is all the information I need to write or in my case to pre-generate and a Python method I have the name. I have the list of parameters I know which ones are required and which ones are optional so I can actually choose that these ones will be positional these one will be keyword and I Have the way how to actually put all those information together to create a URL and send it over to the server the last nice thing about this is It minimizes the effort to maintain it because we stuck it into the same repository as elastic search code bases The same repository where the documentation is and that meant that Updating this just meant that whenever I make a change in elastic search itself whether I add a new API or just add a parameter Inside the same comment or all pull request. I also provide changes towards the Specification and then all the client people all they need to do is just monitor this one directory on github to see all The changes that they need to implement in their client but Again, we we approach that the difficult thing like people need to watch something and do something and whenever you rely on people You will get into trouble sooner or later. Hopefully later, but you will probably get into trouble So that brings us to our last lesson test everything don't trust Just verify everything in our case we needed to verify that all the clients are consistent and That they work well with the server So again, we created our own solution and we created a unified test suite. We again took a machine parsable language in this case YAML and Created a simple simple test suite with a setup and a bunch of actions and a bunch of assertions That enabled the code to run not only against Alas you search itself, but also against all the clients So this is how it looks so this is again a test for the suggest API and you can see there is a setup that will actually Call an action called index with the parameter of index type ID and body and It will then do our fresh so it will make the document available for search and then it will There is one test the basic tests for such as API And it will actually perform the suggest operation and then run to assertions So this test validates that the suggest API is actually capable of correcting our typos and this is a test that is run as part of the Java test suite as part of the integration tests and So that means that it's version specific it's in the same code base So whenever you have a branch of elastic search it can have its own its own test suite just like just like any other tests and Also, all the clients have an interpreter for these tests That makes us sure that we have the same naming. We have the same API coverage We have the same exception handling because we can know just one assertions But we can also run assertions that this should fail with this error code and Just by specifying it once We can make sure that all the clients are consistent These set of tools together Led to that when we decided to develop the fifth client after the original four It only took a few weeks for the for the JavaScript guy to write an interpreter for the for the test suite and the parser for the API specification and Make sure that everything is working as it's supposed to So these were my lessons that that we that we learned during during this process It was it was good times. It was bad times But we we made it through and I believe that the clients are working well for people and now we're sort of approaching the The next stage and that is to actually create a more high-level opinionated clients that will be more helpful to and our end users But also for those clients. We are okay with people not using them So thank you very much, and if you have any questions, I'll be happy to hear them. Thank you So questions Hi, thanks for the talk. Did you try to think about if it was possible to Generate the clients completely Since you already have everything in theory you have the parameters you have the return value so Some clients actually do that for example the JavaScript client. It doesn't actually contain much source code It just actually internalizes that the JSON specification and it generates the method on the fly for Python I actually wrote a script to generate the entire client and I used it as as the first draft and then I edited manually because It's great to automate everything But usually it's okay to automate just 90% and don't try to catch catch it all and just do the 10% manually It's the classic 80 80 20 problem So I started with the generated code and then I filled in all the exceptions All the exceptions to the rule so now when there is a change I run the generation process again, and I manually look at the diff and and see what parts Represent an Actual change and what was just a manual edit that I had to do in order for the API to actually fear more like Python Okay, have you been trying or considering to use the protocol buffers from Google? So Well Yes, we have we have actually considered that as an alternative transport Currently we are fine with just HTTP and JSON though. We provide some alternatives There is an there is an experimental transport with a memcache protocol and Redis protocol and thrift We haven't looked that much longer on Protobuf because the the trade-off didn't seem to be that huge to to Warn that that investment. However, we are still we are still looking for more effective Transports and encoding schemas. So it might still happen It's definitely something that you can implement yourself in a plugin for both the clients and the server Questions no more questions Okay, then that's it. Thank you again. Thank you very much


Let's get this started. I'm really excited to be here to talk about an amazing company right out of Europe It's going from Poland to the world. So let's start it off by talking about Can you tell me a little bit of more about the founder story of brainly and how you got the idea? Yeah, sure. So as was introduced brainly is a global learning community We unite students teachers and parents in solving their academic problems And we help them to exchange knowledge between between each other and The story started with the inspiration was my high school. Okay, so my class was very collaborative We were we are asking questions answering questions all the time And and the foundation for brainly is to move that behavior online. So we build the brainly with the Assumption that's okay having 20 friends in your classroom or 40 friends Depending on where you are around the world is great, but being able to ask your question to millions of other students it's it's even better and Yeah, like every question that gets asked we put it into our knowledge base. So then it's easier accessible for other students and Yeah, that's that's what we do and that's the brain start awesome So can you tell me a little bit about how the platform itself works? So when a student goes on can they just ask all the questions they have Can they just put their homework up and have everyone answer it? How does it work? Yeah, so we have three Types of usage one comes from students so students the typical use case is that there is a pop quiz the next day And they are trying to prepare and they don't understand something or there is one of the homework questions It's really difficult and they don't understand how to do it So then they go to brainly to ask questions and and get and understand the topic But we also have parents and teachers using us So the typical parents use case is that they were asked a question by By their kids and they don't know how to answer so they go to brainly Check the answer understand how to do it and then they go back to their to their kids to explain and the teachers use case is Is quite special because we are really engaging to teachers who are passionate about education So so there are teachers who even after school after after the paid time of their of their day work They still would like to help students a lot And what the brain it gives them is that they can help thousands of students Not only the classroom that they have at school. Wow, that sounds like an amazing community And as was just mentioned before now you guys are you have a hundred million students from all over the world But you started in Poland So can you tell me a little bit about of course you are from Poland? So yeah, why you started there Why didn't you just go to the US or somewhere else in Europe to start your company at first? Yeah, we started about 10 years ago. So That was a bit of a different time in Poland from funding perspective Mm-hmm, and we just wanted to focus on the market that we understand so we knew that this niche exists in Poland But at some point we and we discovered that there is more than a billion students and Every student needs help in learning every day and this is where we would like to be to help them So we decided to enter new markets and we started with Russian speaking markets Okay, and Where we have about 80 to 90 percent of middle and high school students using us every every month And right now we are really excited about India. It's our newest markets growing really fast. So Exciting times. That's awesome So how did you manage that transition from focusing just on Poland to focusing on places like Russia and places like India and even more countries? Yeah, I think the big conclusion was to understand which parts of our product We need to globalize and have one unified a version between all of the geographies But also which part needs to be Localized for each market. So like right now that answer looks obvious because every country has a different curriculum But we were struggling to understand really to put that border between global product that is unified with Localized part of the product. So our knowledge base is localized So which market has its own knowledge base where students ask and answer questions Okay, but the product from look and feel perspective is exactly the same around around the words I think that was the main challenge to re-understand. What is global? What is local interesting? And are there any interesting specific examples about localization in specific markets that surprised you or yeah So for example, India is very focused on test preparation and on and and students in fact are very committed to to learning to get Like to pass the exams in the best possible way because that's the ticket for them to get an engineering degree or medical degree And and what we've noticed is that that use case in our Indian platform is very very popular Markets differ between between each other depending on development of the educational system and Every educational system has its flows and we see that on the daily basis in the way students use our platform Very interesting So from Russia India and tons of other countries now you're moving into the United States Which is one of the most competitive markets for education in the world Can you tell me a little bit about that decision to go to the US and what's what is it like there? Yeah, so what we've seen like us is obviously that like huge market with a lot of spending power Done done online. So I think it pops up on the radar of any any consumer company around the world So so that was that you know that country that we really wanted to enter but But the journey Is is slightly different because we decided to set up an office to to enter the US market We just thought that that's to win the US. We need to We need to focus on our product make sure it's it's polished to the to the extent to like match the competition But in the meantime what we've discovered is that there is in fact quite a lot of products that go kind of like via schools to students Okay, so they approach schools teachers and then teachers refers students to use the product But our model is different like we approach students directly because what we see is that Students are a lot more active if it's their own decision to use the education product Definitely, and then what we see our teachers and parents to follow and there are products that are using that model in the US But they're not that that many in that space specific Interesting, so it's really like reaching out to those students and getting them involved And then they get those teachers and parents involved as well. Yeah, so how do you reach out to the students? Is it word of mouth? How do you build that community? Yeah, so so we we are lucky to have purely organic growth So our marketing budget is pretty much non-existent And our growth model is fairly simple So when a question receives an answer we put it into our knowledge base the bigger the knowledge base gets more people are coming And that means more questions more answers So that's this snowball that keeps growing and And helps more students very interesting and I guess so you guys serve teens, right? How do you really get into their heads and understand what their problems are with education and homework? How do you understand that? Yeah, I think that's that's the toughest part of this because students and teenagers are Are this user group who is changing their their products very very easily? and our answer to that was really to focus on On on our product and keep it simple and focus on one use case and that use case is to Make sure that the learning is more is more efficient Okay, so then when you remove that part kind of being in the fashion type or Or being just a trend that's that this generation of teenagers uses, but the next one may not be using Like was that cautious decision? So so we just try to like deliver the need that every student has and And do it with a with a simple product that they can easily understand and that's our way to stay relevant. So Yeah, we started in Poland about 10 years ago And and yeah, like we keep growing and we don't see any Any pushback from students and I think that's that's kind of that combination of luck and planning And how are you working on a personalization for students? How does that work? Yeah, so so what we are doing is that we have this huge knowledge base So we have about 70 million answered questions in our knowledge base and what we do is that? The ultimate vision is to suggest the right content or the right person to help to any student that is using our products so So so what we are doing is that we are investing Investing a lot into our machine learning team to understand how the content is connected But it's not only from the purely academic perspective But also to help students connect the dots so they can see that okay linear equations can be used in this specific way in my adult life To make sure that they are more excited about learning Interesting so one thing that I thought was really cool when I was reading about brainly was these brainly superstars these students that go in and They answer hundreds of questions and they're kind of like I don't know if influence is a good word But they're just like an amazing example of someone using the platform Can you tell me about these superstars? Have you met some of them? Yeah, like in fact recently I've been to an event in Warsaw and somehow there was a 14 year old boy Okay, I Don't think it's Hello I'm not sure that the mic is working You want to talk one more time?  Yeah, I think the It's not working. Oh well Okay, I Guess it doesn't perfect Thank you so much so the bring me superstars and this 14 year old kid at this event in Warsaw tell me about him Yeah, and and he approached me and asked hey like I read the person who started brain And I said yes and he said thank you because when my parents cannot help me with homework And I really struggle with something brain is there to help me But we what we've done was to look at research and what we've noticed is that Brainy as a product makes students more curious So before that when they struggled with a question They didn't really have a lot of options because they could ask their parents They could ask their friends, but both options are are very limited time-wise, right? But because they go to Brainy and they understand topics they can they they start embracing more tough questions And and then from from the perspective of of asking new questions. They just feel more empowered Interesting cool. So then I want to pivot a little bit more and talk about ed tech in general So what do you see being some big trends and ed tech over the next five years? What are you excited about? Yeah, I think that there are a few things like one space is are all of the products that make the school more efficient So so as I mentioned there is quite a lot of products who go to schools and try to get teachers excited and then students kind of follow the choice that teachers made and That space like is going to help to make the classroom more engaging more rewarding But also to optimize the time of teachers because teachers in fact work a lot and they need to put a lot of extra hours if they If they really want to become Amazing teachers and be considered like that. So my sister is a teacher and I see how much work she needs to do And I've seen her embracing new technologies and and that works really really well for her The second part as I've mentioned before is that is that direct to students products So it's kind of that outside of school space where students can Depending on if they need to catch up with a topic or if they if they are really interested in something They want to explore more and learn more They can they can get there and and get that help So I think these are the two major trends right now and then there is a question of delivery So of a platform and like you have mobile products You have AR VR all of that wheel we reinforce these two big trends, but I think that big trends will stay Pretty much pretty much the same. So within all of those trends. What's next for brainly? Yeah, I think there are a few things one thing that we are really excited about is is our growth in India It's our fastest growing markets really focused on education And I think educational technology space is really strong in India Students are really motivated that they understand that that to progress their life They they need to spend a lot of time learning. It's that it's the same with with parents and And in fact what we see is that there is quite a lot of products already there that we can learn from But what we can do with that with that learnings is to is to make them global so so recently we've we've noticed a behavior of our students in India of our users in India and And then we build a feature for for that use case and we've noticed that it can work very similarly around the world Interesting. So so what we so what we are doing is that because we are present in about 35 countries We can get learnings from each country about making learning more efficient and then we can implement that solution around The world fascinating and what about the US? What's next for brainly in the US? US is is a market where we are where we are growing fast. Like in fact, it's one of the markets where we monetize So we are really focused on on on building a successful monetization story based on our US US traction But yeah, like we would like to onboard more Students more more parents and teachers in the US and that's our main focus for that market very interesting And so as you're going forward, I just wanted to ask if you think there's any Advantages to being a European company or your experience coming out of Poland that's helping you compete on a global stage I think like We quite quickly realize that's to that to build a big impactful business in Poland We need to become global and and that helped us to push our culture to be very diverse and and International very early, okay, so So I think we have about people from 20 different nationalities working on the brainly right now and we and we embrace that That cultural background that we need to understand to be present in so many in so many markets And I'm not sure if that's true for Other European countries, but when it comes to Poland and building that B2C business I think it's a great advantage for us to have So many cultures and try to understand them and build products that that are appealing to to all of the countries Yeah, I definitely feel like that's true of many European startups They have that diversity and that resilience kind of baked into them from the beginning because there's no single one market here in Europe they have to learn how to understand many different consumers from the beginning so definitely feel like that's true So then one vinyl thing is do you yourself go on to brainly and actually answer questions? Yeah, I try to do it as often as possible, but The reality is that questions at high school level are so difficult That's why I totally understand parents who are Struggling with helping their kids. Yeah, and and why they come to brainly because like the level where I answer questions are Middle school questions and I have questions about interest rates and In math, I think that's that's why my sweet spot and this is where I'm trying to where I'm trying to help But high school is definitely Really difficult. Yeah, definitely. I've looked at some of my brother's homework and I'm like, oh my gosh I could never even do this Very cool. So how many questions do you think that you've answered on brainly? I think I've answered in total like Tense of questions and it's hard to aggregate because like brainly as I mentioned is localized per market So I started answering in Poland then I started I've been answering in our US version That's so that's my activities kind of spread between between different markets. Wow international brainly user Okay, and then the final question is do you have any advice for found European founders today who are looking to create a global business? What's a one piece of advice you'd give them? Yeah, I think it's it's important to To embrace what what your background gives you and as I mentioned in our case It was to see that you cannot build a you can no longer build a huge B2C business out of Poland and then that pushes you to to think about So what are my next steps and what's the strategy to really achieve your your ambitions? And then it's about keeping the focus making sure that you You are focused on the one thing or like to maximum free at the at a given time and that you push and push and push Because at some point there will be that rewarding moment that will come and and that's a great feeling Awesome. Well, thank you so much for your time today. It was awesome to hear about brainly Thank you very much and anyone who's good at math go on and answer some questions and help the students out of brainly Thanks so much. Thank you