Welcome back everyone. This is Brian and we're going to continue our journey into Python 3 with communicating with processes. Now, what are we even talking about? Well, we're talking about real-time communications with processes that are running. This is not getting an exit code back. This is taking a process as it's running and sending it information. We can even do this on other machines. That means we can now do multi-processing across a network. Let's open a diagram here and let's take a look. I'm just going to put this rectangle out. This is going to represent some sort of computer. Then in this computer, we can make some happy little processes that are just turning along, living their little existence as processes. They may be totally oblivious that the other processes exist. What we can do now is we can take some sort of communication channel and communicate with that process. That is done internally via sockets. This is networking. When you say networking, you immediately think, now wait a minute, networking sounds like internet. That's absolutely correct. It gets way more complex than that. To overly simplify things, and yes, I am drastically overly simplifying things. We can take another computer and this computer can have its own processes running that are just happily churning away. Oops, don't know what I clicked there. Happily churning away and they are oblivious to each other. They're even oblivious to the other machines. We can take this communication channel and move it to another machine. Now you have two processes on different machines that talk to each other. When we talk about networking, there's some fundamental terminology we need to understand.  A server in this case would be this one right here. It's listening for a connection. The client would be this one over here where it's connecting to the server. You notice the arrow is connecting to the server. Now, that communications can go back and forth, but in this example, we're going to do one-way communications. We're going to cover more complex things when we get into networking 101, which is very, very soon. Also understand that you can have multiple of these. So for example, you can say, hey, I want to communicate with this guy over here as well. And I want to communicate with this one on the same box. And you can make this as complex as you really want it. Now, the caveat being, though, that once you start using networking, you now have all sorts of terminology and issues and things that you need to understand and other issues you need to work with. For this example, we're going to work on the local computer, but I'm going to try to explain the terminology as I go. And there may be some creative Googling you're going to have to do if you don't fully understand. But this series will cover networking in depth. We're just simply not there yet. Okay, the first step in any program, of course, is imports. I'm going to do a little bit of creative copy and pasting here. So we're going to import logging time because we want to be able to stop the current thread. Multiprocessing because shockingly, we're going to be working with multiple processes. And then from multiprocessing, we want to import process. That way we can get that process name. And multiprocessing context, we want to import the process class. We want to add one more and I'm going to actually type this out so I can explain it from multiprocessing. Connection that connection gives you a clue that we're going to be doing some sort of communication under the hood. We're going to be doing TCP communications or this is a TCP connection. If you don't know what that means, don't worry, we're going to cover it in depth in future videos, but just understand that we are now making some type of network connection. From this, we're going to import the listener. And when you hear the term listener think server, this is something we're going to connect to. It's going to listen for connections. And we're going to import the client. This is what's going to connect to the listener or connect to the server, if you will. And from there, of course, I want to configure logging. That way we've got that configuration in there and we don't have to worry about it. It's going to exist and we can now work with it. Now that we've got the basics out of the way, we're going to work on the worker process. And if you're not familiar with networking, this is going to be vastly confusing. I'm sorry, I will try to explain it as I go along. We've worked with processes before, but this is a little bit different here. So we're going to use a function, I'm going to say def and let's call this proc. And for now, let's just say pass that way. The ID doesn't get all mad at me. I want some parameters here. I want a server. And we're going to say, what server are we going to connect to? Now, this is a little bit misleading here because we are going to be the server. So I'm going to say local host. This is part of what's called an address. So an address is a combination of a server, which can be a host name such as local host or an IP address, which you've probably heard that term before, and a port. And this is a number. You have thousands and thousands and thousands of ports out on your system. Now, the caveat being someone else may be using a port. There are what's called well-known port numbers. For example, emails like port 25, FTPs port 21, web traffic's on port 80. So if you try to use a port that is already in use, your program is going to fail. There are more advanced techniques where you can find out if the ports in use, but we're just going to live dangerously and just see if it works. Mainly because we're in newbie land and I don't want to do a three-hour video on networking. And then we need some sort of password. And this is not going to be industry strength password. By any means, we're going to make some sort of byte. And we're going to say password. Never, never, never have a password of password. Somebody will get into your account very quickly. So the basics here is the worker process is going to start and it's going to listen for incoming connections on an IP address or a server name and a port. These two combined make the address. A person connecting needs to be able to authenticate to us or log into us using a password. This is overly simple. It is not the most secure thing in the planet. This is really great for newbies though. So I'm going to say name equals and I want to know the process, current process, and I want to know the name. From here, I'm going to say logging.info. And in case you're wondering just how complex networking can be, it can get extremely, extremely complex. Python has a couple really, really good libraries out there. I should say libraries frameworks out there that we're going to cover. And honestly, it just, it'll blow your mind if your brand new computer is just how complex you can make an application. All right. So we're going to just say that this is started. And of course, I want to know when we're done. So we're going to say finished. Now in between here, this is where the real work happens. All right. At this point in the function where you're doing absolutely nothing but logging. So let's actually do some work. We're going to start listening for connections. We're going to take the server and the port and smash them together in one variable called an address. So I'm going to say address equals server and port. There we go. Now that we've got that, we're going to go ahead and make a listener. I'm just going to lowercase that. And the listener is going to take an address. Simply give it the variable we just created. And we need an off key. The off key sounds big and scary, but really it's the password that we defined up here. Ta-da. This guy. So really it's just a bunch of bytes and do not ever use something as weak as password. This should be some strong type of password, but just for an example. All right. In case you're wondering why I'm not using a strong password, because if I have something like this, somebody's going to go, what are those characters mean? So it's just very self-explanatory to say password. All right. Now that we've got our listener, we need to do something with that. So I'm going to say connection or con for short. And we want to accept an incoming connection. I want to be able to see that some type of connection happens. So I'm going to say logging.info. In case you're wondering why I'm typing this, yes, there are frameworks and libraries of fun stuff out there, modules that you can do that will replace this functionality, probably better than what I'm creating it. But I wanted to show you the foundations here. Connection from, and then I want the listener, not last underscore accepted. And that'll tell us who actually connected to us. Now that we've got some type of connection here, we are going to loop for input. We're going to basically monitor that connection and wait for some specific command from the remote process here. I'm going to say while through message equals, we want our connection. And we want to receive. So we're basically saying, hey, connection, a lock until we have some type of data here. And then notice this is a pickleable, say that real fast pickleable object. We talked about pickle in the past. But under the hood, Python is taking an object serializing it with pickle sending it across the TCP wire and then deserializing it and handing it back to us in the form of a variable that we are going to call message. And now we're going to say logging.info just because I want to know what's going on under the hood here in case something goes kaboom. Name, and I want to say data in, and I want to see the actual message that was sent. That way if somebody sends gibberish, I want to know. Now, you can do all sorts of commands, you can do pretty much anything we want, but I'm going to say if the message is equal to wit, just the string representation quit. I want to say connection close. And what this does is it takes that connection between the client and the server and closes it. Think of it like you're on the telephone with somebody and the conversation's over.  That's literally what we're doing here. We're breaking that connection. There will be no more communication between the client and server once we do that. Then we're going to break out of this loop. And once we get back here, we're going to say listener.close. So now we're going to actually close the server down. Notice those are two distinctly different closes. Here we're closing the connection. Here we're closing the server so no one else can call us. That's like ripping the phone out of the wall and throwing it across the room. All right, so quick recap before we move on. Really what we're doing is we're going to start listening for connections and then we're going to loop or input from that other connected process. It's going to send us variables or I should say data that has been serialized using pickle. Now let's work on our main function. And I'm going to break this down into a couple little segments here. So we're going to go here, main and this little segment, I'm just going to flesh out the template for main here. So we're not going to win any awards for this code, but it will still be cool. Why not? And we can just for the sake of time, Lop in some stuff here. I'm going to grab him. The magic of copy and paste. And now we're going to start fleshing this thing out. We're going to do this in a few different steps. We're going to set up a process, connect to the process, and then clean up after we shut that process down. Okay, step one, we are going to set up the process. And I want to start off with we need an address, a port and a password. So I'm going to say address. And this is where you would put the actual address. You are somewhat skilled in networking. You say, Hey, I'm going to use my buddies machine. And you're going to say, you know, one nine, whatever their IP address is, blah, blah, blah, blah, blah.  They may not have a port open on their router. And I mean, all these other networking things that we haven't talked about are going to happen. So no, you're not going to have Uber hacking skills after this video, but you got to start somewhere, right? So we're going to say local host. Now, when we see local host, what are we really connecting to? That's our local computer. Now under the hood, you have an address that looks something like this. That's called a local loopback. So when you see local host, that is actually the exact same computer that you're currently on. So don't ever really get confused. You're not going out and connecting to another system somewhere. Now from here, we want a port. Now the port is a little confusing. Every host, whether it's local or otherwise has thousands and thousands of ports. Some are available, some are in use. So we're going to pick a port that we're fairly confident is not in use. I'm going to say 2923. There's a much better way of doing that. We're going to cover when we get deeper in networking, but right now just kind of leap of faith that there's nothing running on port 2923. Special note on that. If you get like rejected or denied or port and use, that's probably the number you're going to have to change at that case. Just go in here and randomly change it to another number. I would keep it above 1024. And I'm going to actually put that in the notes. Because anything below this number is what's used as a system port. This would be things like email web services and things like that. So you don't want to go below that 1024. And honestly, I would go above 2000 just to make sure that every computer is going to be a little bit different. And if you have a virus scanner or something like that, it may actually pop up and say warning, this thing is trying to communicate on the internet. Just no, you're not hacking yourself or anything like that. So you would not believe some of the weird questions I've gotten over the years from these kind of tutorials. So from here, we need something super snazzy and top secret. This is just a bite representation of some sort of passphrase. You can put whatever you want, but this password has to match whatever we're expecting over here. That's the auth key right there. So once we've got that, now we can go ahead and actually start the process. So I'm going to say p equal process. And I want the target to be our function. And I want the args. I want that word args to be the address and the port and the password. Now we also want to set this to a daemon. That way it'll, whoops, what did I do there? Oh, good gosh. All right, we want to set that to a daemon. So if we shut this program down, we want everything to shut down with it. We don't want some rogue process sitting there turning away. And let's go ahead and give it a name just because we can. Looks mean, ugly, and scary, but we've covered this in other videos. And we don't need two equal signs. I just horribly screwed that whole thing up. So anyways, we're taking a process class. We're going to say your target is going to be this function up here, our worker process. The args are going to be the address, local host, the port. In my case, I've got 2823, choose whatever you want. And some sort of password. Now you should do a whole lot more than what we're doing in this video. We're just covering the basics. We're not covering like high end authentication or anything like that. So I'm going to say start. This is going to start that process. Logging info. And let's go ahead and say name waiting on the worker. And there's various ways we could do this. But I want to have you understand why I am going to put this to sleep. We're going to sleep for exactly one second. The reason why we're doing this is we're doing two things rather quickly. We are starting a process which takes time. And in that process, we're starting a listener and waiting for connections. So now we're communicating with the operating system and the TCP stack under the hood and saying, hey, pretty please open up a port on this server. All of that takes time. Granted, it happens very, very quickly. But I'm just going to say go to sleep for one second while that process loads up. You can try things like, you know, is alive and all that other stuff, but you're going to run into some weird timing issues where one time it works and one time it doesn't. So sleep for one second because one second in processing time is an eternity. All right, now at this point, hopefully we've got a process that's running and that process is hopefully listening for a connection. We're going to go ahead and connect to that process. So I want to say I need some sort of destination and this is going to be a combination of our address and the port. It's virtually identical to what we did up here. We're just using different variable names so people don't really get super confused. So we're going to take that destination, which is a combination of the address and the port and we're going to say make a connection object. That connection object, we want to do something super, super fancy with. We're going to say we want to make a client and we want this client to connect to that listener on that destination, which is a combination of the address and the port. We also want to give it the off key. In this case, it's going to be that password, that byte representation of a passphrase that we set up here. This has to match. If it does not match what we started our process with, we're going to have a bad time. So what happens under the hood is, well, magic at this point. We haven't really talked about networking and TCP, but what's going to happen is the computer is going to go out and it's going to try to connect to that remote listener using the destination address and port. If you have any problems, this is where you're going to start getting errors. So we have a process. We have a client. We've connected to him. Now we need some type of command loop. And this is very simple. I'm just going to ask the user, hey, what do you want to send to the server? So I'm going to say, while through, I'm always envious to these guys that can like type and talk at the same time because I am not that guy, to be brutally honest with you. I'm going to say input.  All right. So we're going to hard return line feed, hard return line feed. In here, I want to say enter a command or type width. We're going to go ahead and take that. I'm going to say strip because we want to make sure there isn't any white space after that. Python is not a big fan of white space and neither am I. And then I'm going to say logging.info. And we want to know what we're sending. So I'm going to say F name, command, and then our actual command so we can see what we're sending out to the server. And then we want to actually send it. So I'm going to say connection.send. And we're going to send that command. That's just a string. So it's nothing fancy. It's just going to shoot it out. Now, networking is actually highly, highly complex. If you're ever super bored one day, take a networking class. You'll be overwhelmed very quickly. But there's a lot of complexity under the hood. All we need to know is that thing sent without a problem. And then we want to see if the command was quit. Then we're going to go ahead and pull rake out of this loop. And we're going to move on to the next section, which is clean up and shut down. All right. The last step, we are going to clean up and shut down. And this is well dead simple. Some say if the process is alive, then I want to try to gracefully terminate that. So we're going to say logging.info. And I want to say name, terminating, worker. That always sounds so harsh. We're going to terminate something anyway. So we're going to say connection.close. So we're going to try and gracefully close that connection. I'm going to go ahead and say time.sleep. There is vastly better ways of doing this, but I'm just showing you the very newbie way of doing it. What's happening under the hood here is if we inter quit, we're going to break out of this event loop. So we want to, well, wait for that process to terminate. So if it's still alive, we want to close that connection down, give it a moment and then, you know what? If it's still going, no more Mr. Nice guy. We're going to completely terminate it. From there, we're just going to say P join. Join back into our main process here. All right. So a lot happening here. This seems complex. It's because it's bigger than anything we've done in this series so far. But really what we're doing is we are starting a process. That process is going to run and it's going to have a listener inside of it. That listener is going to accept connections. That connection allows us to send real time commands to it. We can take those commands and read them and act upon them. If we send quit, it's going to close that process down and terminate our program because it's going to exit out of our command loop here. All right. Moment of truth. Let's see this in action here. So let's move this up and let's run this. So main process started main process waiting on the worker. The worker has now started. We have a whole new process running. And then here's our command loop right here in our commander type quit. And you can see the worker connection from this. Now this is a little confusing right here. What is this guy? This is why I wanted to split this out into a different section. So local host, remember how I said it is one two seven zero zero one. That's your local loop back. So basically what we're saying is connect to yourself. There's something in the background called name search order or DNS. And basically I'm going to drastically oversimplify this. It's going to take a name and turn it into a number. And under the hood, it's smart enough to know that local host is this number here. So when you go to like Amazon.com or Google.com or Apple.com or something, there's some sort of naming convention going on there where it's doing a name lookup. 99% of the time nowadays that's DNS, but it's basically saying under the hood translate a name into a number. Now you notice the port is different too. We have port 2823 and here it says 6952. Every source in every destination have two different ports. So our source is this guy right here. What happens is under the hood, the operating system tells Python, let me go get a free port that you can talk on and it hands it this number. It's a lot like me handing you the telephone saying here you go. It's not your phone and you have to give it back. And if you don't, I will come find you because I really like my phone. However, the person on the other end aligned the actual listener. They already have their phone and they're ready to go. So that's why you have two different port numbers. Just understand you can't have the same port number on the same machine at the same time or you're going to have very big problems. All right, back to demonstration land. We need to enter a command we can just simply enter cat or whatever you want. And you see the main process command cat and then it sends it across the network and the worker says data in cat. That is extremely cool. Now you may be wondering, because this is all on my local computer, is this actually going across the Internet? The answer that is no, your TCP stack on most operating systems is smart enough to know that this should be self-contained because it's all on the local network here. Let me see if I can highlight that. If it's local, it should never actually go out to your network. But you could very easily run this script out on another network machine with some minor tweaking and get it to load up just fine. Now I'm going to go ahead and shut this down. So I'm going to say quit. And you notice how our main process command quit. We're going to terminate that worker. The worker's got a data in of quit. And then the worker finished. There is a slight delay here, which is why I put all those times sleep. If you take those out, you're going to start getting some weird errors. And then finally, our main process is finished. This is pretty complex, but Python makes it extremely easy compared to other languages and other frameworks. The main takeaway here is that you can start a process even on a remote machine and communicate with it in real time. I hope you enjoyed this video. You can find the source code out on github.com. If you need additional help, myself and thousands of other developers are hanging out in the Voidromes Facebook group. This is a large group with lots of developers. And we talk about everything technology related, not just the technology that you just watched. And if you want official training, I do develop courses out on udemy.com. This is official classroom style training. If you go out there and the course you're looking for is just simply not there, drop me a note. I'm either working on it or I will actually develop it. I will put a link down below for all three of those. And as always, help me help you smash that like and subscribe button. The more popular these videos become, the more I'll create and publish out on YouTube. Thank you for watching.


2020 is the latest installment of an annual series that documents antibody therapeutics approved in the US or EU, those in regulatory review, and those in late-stage clinical studies. As of November 2019, five novel antibody therapeutics had been granted a first approval in either the US or EU, and marketing applications were undergoing review for 13 other antibodies. Additionally, there were 79 antibodies undergoing evaluation in late-stage clinical studies, with 40 of them being treatments for cancer. The biopharmaceutical industry's pipeline of antibody therapeutics is robust and should provide a continuous supply of innovative products for. This article was authored by Helene Kaplan, Renalini Merlederan, Zeta Schneider, and others. We are article.tv, links in the description below.


So we have a couple of minutes for questions for really the three outstanding speakers from this morning. So if there are questions from the audience, please come forward to the microphones. For our speaker, and related to the TCGA efforts, how our speaker comments on the like for data, for sample preparation and data preparation, and how mostly this is like a web lab researchers involved in this. And I'd like to hear the speaker's comments on how computational side people are involved in the sample preparation and data preparation. And also on the other hand, is for when we design new computational software and tools like computational people, and we work on this, like how the web lab researchers are involved in this. The second question is specifically for our third speaker and related to the subgroup as a molecular subgroup of the cancer. And I'd like to know specifically what kind of computational software you use for that. Thank you. So I'll go ahead and take the first question, which was I think how basically the clinical side works with the computational side in this project. So TCGA, the way we run the pipeline is the front end of the pipeline with the tissue source sites. The tissue source sites are first involved very early on when we established a disease working group. So when TCGA determines that we're interested in providing this resource on a given tumor type, we bring together experts that include pathologists, oncologists, biologists that study specific organ sites. You know, if it's thyroid, we bring an endocrinologist, et cetera. So the clinicians come in from the very first step. They provide us all the guidance we need in terms of determining what our inclusion and exclusion criteria are, et cetera. And then that group works with us to acquire the samples. That same group is then brought back at the end when we start doing the analysis. So the TCGA funded investigators in terms of our data generating centers and our primary data analysis centers are not necessarily experts in every single tumor type that we study. So we bring in those external experts back into the loop when we start doing the analytical phase. Now, we also have every center does their own analysis on their own data set also. Plus, we have these seven genome data analysis centers. And each one of those groups actually works with clinicians generally on site at their own institutions to make the tools more clinically relevant for the disease-specific parts of the data that they put on their own specific portal. So, you know, MSKCCCBio, which is also, I believe, having a workshop this afternoon, they have their own computational suite, essentially fire hose at the brode also does the same. And so a lot of these groups also have their own different interpretations and their own different algorithms that they're using. And depending on what the right tool is for the question at hand is when we bring those specific analysts in with the specific disease experts. And again, right now it's still one at a time. For the pan-cancer group, it's a bit more of a challenge because we're going across many, many different tumor types and the clinical questions are obviously much more difficult to generalize. And I'm going to turn it over to Lee for the second question. Just real quick, there's many, many ways to use profiling and make cancer subgroups. So we happen to use a statistical model based on Bayes' theorem where we estimate based on a signature of roughly 100 genes, the probability of being one or the other type. So it's standard stuff. I have a follow-up question for Dr. Stout, too. I'm not sure how plugged into the current regulatory environment you are with diagnostic testing and thinking about clinical trials and stratifying based on genotype or gene signature and how that interacts with new and proposed rules that would need to be in for testing where testing would need to be validated before the trial. And so we're talking about, I think, a real challenge going forward. That's thanks. I'm right in the midst of that, unfortunately. It's a difficult landscape developing the diagnostics and genomics. We're trying to do it for the distinction between ABC and GCB. You could just as well try it for the various mutations. I think the paradigm is that you, for these targeted agents, you need a co-diagnostic that's developed along with the drug, and they then are approved simultaneously by the FDA. To do that, because by definition you will be changing clinical care, the bar for that quality of that diagnostic is extremely high. So it's going to challenge us to go beyond what we consider pretty good in the lab to something that is really going to change whether someone gets treatment A or treatment B. So thanks for bringing that up. And it's not going to be easy, but an important struggle. I think we're going to need to take the next questions offline because we will need to move forward to our next session.