Hit me OK. Lots of echo. OK. Hi, everybody. Julian, that's me on the internet. You can find me in various places. I'll try and share the slides afterwards. And I hope to leave plenty of time for questions in the end, although feel free to interrupt me as we go. So I work for a company called Magnetic. Just a 30-second background. We're in marketing tech. We use PyPy. That's why I'm here. Usually they send me to fun international places. It's actually the first conference I've ever spoken at in the US, I'm pretty sure. But it's wonderful. And like everyone else, we're hiring. So if you want to talk to me about work after this, feel free.  Whoops. So as I talk about PyPy, but just for some context, we'll kind of start at the beginning. Maybe it would be worthwhile just to ask, how many people, if I say, can you tell me what the definition of a compiler and an interpreter is? How many people would be able to answer? OK. Small number, maybe a little less than half. So for those of you, this might be a little bit slow, but we'll start from scratch. So yeah, just for some background. So programs, right? So what's a program?  So whatever it is, roughly, it's that.  I apologize, that's quite small, but hopefully nothing else will end up being that small. So it's a program, I think in the case of Chrome, it's like a bunch of C and C++ stuff tied together with a whole bunch of other stuff. It's a thing written in some language. You give it some input, it produces some output. Getting kind of down to as far as we can go, like CPUs execute machine code, which is basically a bunch of binary. You pass through machine code through the CPU and it does what you tell it to. Side note, just for fun, potentially the CPU is taking that machine code and turning it into a bunch of other stuff and not actually executing the exact x86 assembly, let's say, that you're giving it. So at the end of the day, whatever your program is written in, it has to get down to whatever instruction set your CPU is going to execute. And so for our story before, where we have a program, at some point, whatever language your program is written in, you have to turn it into something that is just exactly that, like a bunch of machine code that your CPU can execute. So you still have your input, you still have your output, but you have to have some sort of machine code that you're going to run, no matter what you started with, Python or whatever other language. You have to ultimately execute what your CPU knows how to execute, which is, for most of us, like x86 assembly or whatever. So just a side note, because we got here, so interpreters and compilers, a nice sort of blurry world between the two of them. But it's relevant for Python and it's relevant for us. So Python's an interpreter. We'll talk a little bit about how it works, both in the case of Cpython and PyPy. But there's kind of a nuanced difference here between an interpreter and a compiler. When we say compiler, we'll talk about a thing that takes a piece of source code and turns it into machine code. Compilers that compile to other targets, but that's what I'll mean. And by an interpreter, we'll mean a piece of software, a program that takes some source code and turns it into something intermediate that then executes on some other higher level machine. We'll talk about some more in a minute. But Python is, in that sense, an interpreter. It's a piece of software that takes Python source code and executes it without first turning it into machine code. So you don't take Python and turn it into single binary. It's not Rust or C or any of those languages. So specifically, I'll start talking a little bit about Cpython. So Cpython is, for those of you who don't know, Cpython is what you're going to get if you just type Python and don't otherwise know you have something else. So if you downloaded something from the internet or from your package manager or from wherever else, you're getting your software. Cpython is what you got. And the reason why it's called Cpython we'll talk about in a minute, but it's basically written in C. Here's your answer. And it's a program. So back to our original definition of what a program is, it takes some input and it produces some output. In the case of Cpython and the case of interpreters in general, the input that it's taking is the source code that you wrote. So you type some Python into a file or someone gave you some. That's your input to Cpython. And the actual process that Cpython goes through to actually execute it, there is a step in the middle there, which again comes back to our same conversation from before. So Cpython takes the source code that you gave it, turns it into some byte code, which we'll see in a minute. And then it takes that byte code and executes it on a VM. And a VM is basically, you can think of it as a virtual CPU. So our actual CPU, we make believe, let's say, it takes x86 assembly or whatever assembly language your CPU takes.  So it's kind of a giant loop. We'll talk about it in a minute. And that giant loop takes an instruction set. And that instruction set is basically going to be used to execute our Python code. And if you look at Cpython, if you've ever pulled apart the Cpython source, there's a file in the Cpython source tree called like C of L dot C. And in there is basically a giant, I think in Python 3, it changed a little bit. But in Py2, it was basically like a giant case statement. And each of the cases in the case statement is like one of the byte codes that the VM accepts. And so for every, Gabe, you have a password? We're about to find out. I'm glad you do. If you didn't have a password, I would yell at you. There you go. So in Cpython, the byte code instructions that we're about to see get passed through this loop, which basically just switch cases over each of the byte codes that flows through the interpreter and executes them one by one. And if you look in C of L dot C, you'll see implementations for each of the byte code instructions that we'll see. So I'll show you some examples in a minute. As soon as we get back plugged in here. Any questions so far? Good.  So I would assume a lot of you have probably seen both halves of this. But again, just for background, we'll go through both. So on the left here is some Python source code. Kind of a strange function, but just for a bit of demonstration. So it takes two numbers. If x is bigger than y, then it adds them. Otherwise, it subtracts them. Why? I don't know. So there's this thing called the disk module, which you might have played with before. Basically, it's a module that can take a bunch of different kinds of objects and show you human readable and not so human readable lists of the byte code that that code object will actually execute. In this case, the functions code object. So if we pass in the function, we can kind of read this from the top. Can I ask you to put this on your desk? OK. So if we just read this from the top, it's actually not that hard to read. It's somewhat typical of kind of like high level VMs. So load fast is basically a byte code instruction that loads a variable. So we're going to kind of compile this in our head and see how that compares to the byte code on the right. So if you think about how this function is going to execute, the first thing that it does really is it needs to know what's the value of x and what's the value of y. And those are variables that were kind of like passed into this function, so it needs to basically look them up. Compare op is basically an instruction that means take the two things that are on the stack, which I haven't defined. But basically, you can think of a stack if you don't know what it is already as basically being the place where we're storing the stuff that we're processing. So compare op means take the stuff that's on the stack, compare the two of them. This one takes an argument. So compare op 0 means do less than, which is what this code is trying to do in reverse, actually. And then pop jump if false means this byte code told us whether x and y, whether x was less than y. And if that was false, then we're going to jump to instruction 20, which if you look at the line numbers down here, 20 is going to be the bottom of this loop, which corresponds to there. And so I can kind of read through the rest of these. But I hope it's somewhat clear so far that what's happening here is there's basically some Python source code. It's been turned into something somewhat resembling assembly, if you've seen it before. And each of these byte code instructions kind of like is instructing this VM in C Python to execute a bunch of slightly lower level, but still fairly high level instructions. You're going to run the slides, good. So C Python, again, it's written in C. It's a byte code compiler and a stack-based VM. That's a bunch of words that mean it produces the byte code on the right and then executes it on top of a stack machine. And then I'm being slightly not fair, but roughly C Python's design sort of prioritizes simplicity of implementation. So the C Python implementation is basically trying to be fairly simple and fairly accessible. Actually, from what I hear, I'm not a C programmer. From what I hear, if you read it, it's half decent C code, fairly well commented, and et cetera, et cetera. But those byte code instructions that we had on there, they correspond to many potential actual machine code instructions. So we had that load fast there, which is like, hey, give me the value of some variable.  Maybe I need to go look that up in some array of where I'm keeping all my values of variables. So you need to implement that using some lower level language, which we said in the case of C Python is C. And in order to actually implement that byte code instruction, which is tell me the value of this variable, there might be potentially more than one assembly instruction or machine code instruction that actually needs to execute in order to perform that operation. And so in the general case, you have on the left all of the byte code that your C Python VM needs to execute. And on the right, for every one byte code, there's potentially n number of assembly instructions that might actually need to execute in order to run that one piece of byte code. Now I'm being slightly disingenuous when I hit the next slide and I say, OK. So if I done slow, right? Because for every byte code instruction that you want to execute, you have to execute, let's say, 30 machine code instructions just to execute one piece of byte code. Whereas if you're using some sort of compiled language, you would sort of get the minimal set of assembly that you needed to execute because there wouldn't be this middle layer. And that's kind of the punch line in sort of this background information is like ultimately when you have a VM, there is a middle layer in there. And the middle layer is the thing that's executing your byte code. And so if you get rid of that, you're faster, right? So that's a bit of a lie to begin with. And I don't actually mean to tell it in this case. So because of all sorts of fun complications and even some non-complications, sometimes actually a VM tends to be fast because of branch predictions and cache missing and all sorts of other fun low-level things. So I don't know about those things. You can probably talk to people who are more low-level programmers than I am. And they'll tell you about them. But don't assume immediately that just because you have a middle layer that it's going to be way slower anyways. Sometimes regularity is good for your computer. But even besides all that. So no, we're not saying Python is slow. First of all, in the pedantic sense, we've just been describing Cpython so far and we're about to talk about PyPy. But second of all, if you think about what we've talked about so far, all of that applies when you're actually executing computation. So byte codes are like, I need to compute something. If you're not doing computation, you're fine anyways. If you need to wait on the network, you need to do something else, you're going to be fine. The kind of basic point that we were making where every piece of byte code might correspond to more than one potential machine code instruction and apply if you're not actually executing byte code. So certainly don't draw the conclusion so far that Python is slow, but we can do a little bit better in terms of just raw CPU utilization speed. So why can't we do better by just doing the normal thing? The normal thing being, let's say, you know your C. How does a C compiler work? Again, it takes the source code and produces some machine code. So why can't we just do that for Python? Why can't we just take the Python source code and produce some machine code in the same way as other compiled languages might do? Why do we need any sort of novel solution? Let's just do the same thing that everyone else is doing. So it turns out that that's somewhere between very hard and nonsensical. So take a function that looks like this, for example. What assembly do you want to compile this particular function to? So if you hit Next. So there isn't assembly instruction that does add a bunch of registers. So this assembly instruction machine code will basically take a couple of machine ints, I think, into registers and add them together, put them somewhere on some other register, I assume. So there is an assembly instruction for adding two integers. But that Python code doesn't add integers. It adds whatever you gave it. So x and y in that function, you pass in two integers. It's going to be integer addition. You pass in two strings. It's going to be string addition. You pass in two random custom user-made objects. And it's going to be whatever those objects say addition means for them. So that's just a function on its own. If you're a compiler and you see that function, how do you know literally what do you generate for what you're seeing in front of you? We'll talk a little bit later about some existing specialized compilers for Python that actually solve that problem in somewhat obvious ways. If in maybe some specific case, you say, OK, but I'm using this function for integers, and then you know what to generate. But in a general case, this assembly code and the Python code on the last page are just nowhere near each other. The Python code is like I add two objects, and they tell me how they want to be added. And this assembly code is like I have two machine words in two registers and add the two of those. There's just no way to reconcile the two things. And it gets even worse if you have something like this. This function doesn't even... The types of this object are on some server somewhere. Whatever... If that JSON object has two ints in it, then this is integer addition. If that JSON object has two strings in it, it's a string addition. You can't hope for a compiler to be able to tell what to do with the types of those two things. So we're in a bit of trouble here. And I don't mean to kind of set up things as like types are the only thing that we're gonna run into trouble with, but certainly it's a big hurdle to get across. It's like this function, I don't know how to type the objects that we're gonna be working with, let alone know how to convert that into some sort of machine code that I'm gonna be able to compile into.  So the punchline is PyPy, and we'll talk about that in a minute, but certainly before PyPy exists, and even to some extent in some circles now, the old answer was like, okay, so you need to change it in some way. So your program's slow, you wrote it in Python, change it somehow. And there were sort of like three, at least three paths, these are kind of the most obvious ones. One of them was like take a piece of it, rewrite it in some other language like C usually, and then use let's say the C Python C API, which is a way that you can take a bunch of random C code and kind of like use it from within Python. So go take your Python code that was slow before, write it in C, then of course you are kind of like statically typed, so you do need to say what all the types of your stuff is.  So your code still behaves the same, but you changed it to basically be written in some other language and then plug that piece back in. And there was an okay solution. C Python has API has a lot of problems, and again from people smarter than me, it has a lot of problems that are quite hard or impossible to fix in terms of like a compatibility standpoint, like the C Python API exists and people use it, and a lot of things in there are kind of questionable, I think, but it certainly an option, it certainly works. Cyton was another option, Cyton for those of you who haven't used it, it's basically if C and Python had a kind of bastard child that would probably end up looking like Cyton, again in that kind of like, Cyton's a good thing, at least in the general sense, like, but that really is what it is. It's basically like it looks like Python with a bunch of C sprinkled through, and there are some additional mechanics that like give you tighter interrupt, but basically it's a thing that kind of looks like Python with a bunch of like types mixed in and some other functionality to like import stuff from C if you actually feel like doing that and a couple other things. And then that thing ends up getting compiled into, again, an extension module that you can load. So this is sort of like using the C Python API as a bit more portable in some sense, and it looks a lot more like the Python that you started with, so in some ways this is changing your code less, and certainly a lot of people do still use Cyton, and from what I hear, some people actually tell me that they have quite a lot of reason to keep using Cyton. So that was certainly another reasonable avenue to pursue.  And then of course kind of like the most drastic or easy solution depending on your perspective is like just wholesale rip out a portion of your program, rewrite it in some other language, and communicate back and forth with some like inter-process communication. So like your program's too slow, great, like take this section of it that's doing computationally intensive stuff, move it into some language that's more suitable for computationally intensive stuff, and then communicate back and forth over whatever method you'd like. So these were the three historical choices. Again, the punchline here is PyPy, but kind of the thing that's getting us to the punchline is a JIT. So a JIT, Justin Time Compiler, is gonna be our way, PyPy's way of basically fixing some of the kind of troublesome problems that we were running into before about like how do I compile programs that I don't know enough about. Justin Time Compiler's the Justin Time being basically at runtime. So while your program is running, it's also potentially doing compilation. That's kind of like the piece of the puzzle that's like okay, you don't have enough information ahead of time to know what you might need to generate, but maybe at runtime it'll be better. Maybe while your program's executing, it'll be easier to figure out what I need to know, and maybe even at runtime when I have more information, I can do even more intelligent things that I could have ahead of time. So we're kind of partitioning the world of compilers now so that we had our languages that we were discussing before, which are kind of like you take a compiler, you plug in the source code, out pops machine code, because that's what you need to execute. Into this world now we're like, okay, we can't do that ahead of time. We're gonna do that at runtime. We still wanna end up with some machine code because that's what's gonna end up being fast. We're gonna get rid of that middle layer, but we're gonna do it at runtime when we have enough information. And so that's what PyPy is basically. It's many things, sort of the most pertinent one for this discussion is that. So in kind of like correspondence with what we were talking about before, it's written in a language called Rpython. I'll mention a little bit more about that in a minute. Again, it's a program, so it has to be written in something, right? So it's a program that takes in a bunch of input, again being Python source code and out pops, whatever output your source code actually performs. So it's written in Rpython, very similar implementation to Cpython at kind of the high level. It again has a bytecode compiler, which means it takes your source code, turns it into some bytecode and executes that bytecode on a VM. The difference is gonna be that every once in a while it decides not to do that and to do something entirely different instead, which will be like produce some machine code and execute that instead of executing whatever bytecode corresponds to your source code. And most importantly, it interprets the same exact language. So we're gonna kind of get exactly where we wanna go here. Before I kind of specifically say, so Rpython, just another word on what Rpython is, the R stands for restricted. It's a language. It's statically typed and it's amenable to being like annotated with types as part of like the translation process. And it's also part of this like larger tool chain that can be used to write other interpreters other than PyPy. But it's basically Python. It's Python with static types or slightly more kiddingly what the PyPy guys basically say is like, it's this thing that accepts whatever subset of Python the translation tool chain doesn't error out on. So it's roughly Python with enough information to be able to statically or to be able to pre-compile just the interpreter part. So, yes. So the last sentence was Rpython that PyPy is written in is basically a subset small enough of Python that it can be translated into machine code. So it has none of the problems that we mentioned before like you know or it, the tool chain knows enough about what the types of everything is to produce machine code ahead of time. And roughly that's the important part. So, you run your Python code with Cpython, assuming that's Cpython, you run the same code with PyPy and sort of the most important part on this avenue is like, okay, we finally, we didn't have to actually change anything in my code. Like I wanted stuff to be fast but I didn't want to learn some other language and PyPy is kind of promising to be exactly that.  You didn't need to do any extra work. So, Jits were kind of like the vehicle for how we got enough information. The philosophy is something like this. So a lot of the trouble that you run into in Python boils down to stuff like this where you can do a lot of stuff but some of the stuff that you can do is kind of unlikely in like a probabilistic sense or maybe not quite that scientific but like in Python you can do a lot of crazy dynamic stuff but you're very unlikely to actually be doing that crazy dynamic stuff and so if you can kind of like make some assumptions that nothing crazy is going on you can kind of like get rid of a lot of the overhead that you would have otherwise had to have dealt with from just kind of like what the language allows you to do perspective. So we spoke about one example before about like just passing in whatever objects you feel like. If you had a program where kind of like imagine as like an external observer which is really what the jit is you're kind of watching this function being called and you see it like 20,000 times being called with two integers. Now it's very possible for your program all of a sudden to start calling it with a bunch of strings instead but from a probabilistic point of view if you see it like 20,000 times you're gonna say you know what probably this is going to be called with integers if it turns out something else happens you'll need to deal with that situation because of course you can't sacrifice the program doing the right thing to make it fast but at least from kind of like a optimization standpoint or like a what can I assume about this program it's probably reasonable to assume that what you're seeing so far is what you're gonna continue to see until you see otherwise. Same thing here for just like literally attribute access. So attribute access is very expensive in Python. See Python. Why is that because like if you want to attribute if you want to access some attribute off of an object you've probably seen like most objects have dictionaries that actually map their attribute names to the values and you can't get rid of that dictionary really besides slots. You can't really get rid of that dictionary because someone can come along to your object and just add random attributes to it. Like your object didn't have an attribute called like blah or whatever. Someone can come along at some point in the program and just plant another attribute on your object and the object needs to keep track of that additional attribute so that like everything needs to know like where's that attribute living in memory. You can't just kind of like allocate a bunch of memory and say like here this is all I need for this object and I can just kind of look up attributes by just indexing into that memory.  So even something as like completely simple as accessing an attribute in CPython like other than slots you need to actually do a hash lookup to actually figure out you know what the value of that attribute is and that's expensive. Whereas if we can make the assumption that we made before and we say like you know what some things actually the language allows you to do but they're not likely to happen so an object like changing types. Like there was a function example. There was a function that every time we see it it has the same types going through it. Yeah I mean it's possible for all of a sudden new types to start coming through but it's let's say unlikely or changing attributes. So classes have some or instances have some number of attributes. It's possible for all of a sudden that instance to have like a whole bunch of different attributes than the ones that it had before but it's not very likely generally speaking objects keep their attributes over their lifetime not necessarily attribute values but at least like the set of attribute names. At least that's constant enough over the lifetime that maybe we can make an assumption about that. And if it's not true there'll be mechanisms in PyPy which I won't discuss to kind of like guards to kind of like make sure that we don't do anything terribly wrong but we can kind of make this assumption try and make things fast and if they turn out not to be true then we'll have to go back to this low root. And sort of the other key piece here is like most programs spend a lot of their time in a relatively small number of places. So if you have like one loop where you're doing the majority of your computation or a bunch of loops where you're doing the majority of your computation, relatively speaking they're probably not like, you know your program probably doesn't spend most of its time in a full 50% of its source code. It's probably somewhere in a bunch of kind of localized places and so what this means combined with what we said before is basically if we can just kind of hit those places and compile them at runtime through the JIT we're gonna basically speed up your entire program because that's where your program is spending most of its time. So all that was kind of section one that was hopefully enough background information just to see what the general strategy is. I'm gonna transition to talking about stuff that we've learned along the way at Magnetic. We've been running PyPy in production at very large scale for probably about three years now. So I'm gonna talk about a bunch of stuff which shouldn't be terribly surprising but just some of the stuff that we've kind of had to deal with along the way and it's some of which I hope is fairly expected but any questions on sort of the first half of things? Okay, I'll be more time at the end. So just in a general sense it's probably most pertinent to talk about like our largest app which we run on let's say 100 servers does like 400K or 500K QPS and just from like literally taking the word Python and replacing it with PyPy just like magic 30% speed up without doing any work. And of course I mentioned before that PyPy is only gonna help you if you're a CPU bound or if you're doing a lot of computation this app isn't CPU bound which is why that number isn't way larger. So this is like a web app you can think of. So it has like a portion of what it's doing that's kind of like CPU intensive and that part of the request will end up getting sped up but overall it's like it's a web app. Not a client facing web app but it's a web app. And so 30% from an app that's basically spending a quarter of its request doing CPU bound stuff is just like literally money in the bank. Okay, so sort of the first thing you'll probably encounter assuming you're convinced so far. So if you do switch to PyPy the first thing you'll probably encounter is like, okay so are all of the libraries that I wanna use actually compatible? So it's actually a decent story for that now in the sense that like there's a website it's packages.pypy.org the URL isn't on there I apologize but it shouldn't be hard to Google if you don't remember packages.pypy.org. So what it is is basically a list of the top thousand packages on PyPy very confusingly not PyPy. So on PyPy, listing of the top thousand packages and then whether or not they're compatible with PyPy actually don't recommend you check it out because I can give you the summary right now. 99 or probably about 95% are and the ones that aren't are things in the kind of NumPy ecosystem other than NumPy itself which is undergoing porting. So like a large majority of the stuff that's on that list is gonna be of the 5% or however many percent is gonna be stuff in that ecosystem so we'll come back to in the end. Otherwise the large majority of things on that list are just green so I would say at this point in PyPy's life cycle just assume that things are going to be compatible other than a specific list and basically again the scientific stuff is certainly one section and certainly needs addressing and the other stuff is going to be libraries that use C extensions are gonna be either iffy or slow because the C API is C Python's C API and so PyPy emulates whatever parts of that it can but even the parts that can't, sorry there are parts of it that it can't so popular examples of libraries there if you're using MySQL Python which I would recommend well first of all they try not to use MySQL but even if you are don't use MySQL Python if you can it's quite old and even if you're using MySQL there are newer libraries out there but if you are it does have a C extension I don't think it will install on PyPy at all. Other libraries I think the Postgres library for Python Psycho PG2 I think also tries to compile a C extension so there are a couple of examples like that but generally speaking for those kind of libraries that have C extensions either it will not use the C extension on PyPy or there are kind of like equivalent libraries so like the Postgres one there's a Psycho PG2 CFFI library which is basically a drop in replacement but certainly like are all of the libraries I'm using compatible with PyPy is the first thing you'll probably have to check if you're porting over an existing thing and again the answer is basically if they're pure Python yes absolutely if they're not either look for pure Python replacements or it might work anyways because of the C Python C API but it's kind of a jarring transition to make when you're first using PyPy because what you start to look for is the exact opposite of what you looked for before because now every time you see a read me you probably want to check and look that it says pure Python which means faster on PyPy rather than what you were looking for before which certainly tends to happen in read me is where they call out hey I have a really fast C extension that I use to actually do computation that's going to make your program slower unfortunately. Other things that you'll probably encounter so this didn't happen to our own code but you'll find lots of programs with bugs in it unfortunately and bugs in a specific sense here so unfortunately this code has a bug and a bug other than the fact that I didn't define process here like this code leaks a file descriptor so it opens a file reads it and then never kind of like closes the file you won't notice this bug on C Python because C Python has a reference counting implementation but on implementations of Python including PyPy but not only PyPy that don't use reference counting this will leak a file descriptor which will maybe eventually get closed but if you're doing this in a big loop and you try opening a whole bunch of these like your program's gonna crash or sorry your program's gonna run out of file descriptors thankfully our own code didn't have I think any of these issues but you will run into if you try running we were on DevPy under PyPy and it had one of these issues recently so like other people's code that you run into might potentially have these sorts of issues and you kind of will have to know to look for them if you see a program run out of file descriptors your first guess when you're using PyPy is like okay someone forgot to close a bunch of files and is relying on kind of like C Python specific behavior there which again the same thing will happen if you run this code under like Iron Python or like Jython or any of the other implementations of Python that don't use reference counting kind of the next thing that you'll probably run into maybe the first thing for those of you who are like Ops see people in the room is handling provisioning, excuse me so if you go on the PyPy website they do have downloads you can like download PyPy binaries unfortunately if you're running like Linux the world is kind of sad when it comes to binaries in the sense that like the binaries that are available are kind of like dynamically linked to some system libraries so when you download a library from PyPy.org if you're on Linux you need to be on the distribution that it was actually compiled for otherwise it's unlikely to work they do have some like semi-portable versions but what we do is basically we use so that portable PyPy thing there which if you're on Linux and you want to deploy PyPy I would say just go straight here even if you are on one of those other distributions it's just easy we basically just plop portable PyPy everywhere and what it is is basically like it's a tar ball with PyPy inside and the PyPy in the tar ball is kind of scrubbed in a way that it actually will run kind of portably so we run CentOS and kind of the PyPy release cycle is fairly fast and generally speaking you do want to kind of stay as current as you can because they do make kind of very significant improvements as time goes on and so like he basically puts out portable PyPy tar balls like a day after the release is done so we kind of just have some provisioning scripts that basically will go off grab the newer version and just like kind of upgrade the whole infrastructure to whatever next version kind of completely portably at least for our sake. There are some issues this one I pulled kind of selfishly so certainly like again the world of Linux is rough when it comes to binaries so like you might notice things like this which is like it statically links against open SSL and like because it does that like there are kind of hard coded paths that it's expecting I think this is an open SSL problem more than a portable PyPy problem but like it won't be able to find your system certificate bundle which basically leads to things like this where like we don't actually do any SSL from within our server apps but if we did we probably would have noticed this sooner which is like it can't find any certificates which means you can't verify any certificates that you might be receiving and so you have to know to kind of like set an environment variable so that when you run that portable PyPy distribution it knows where to find the certificates on your system. So there are certainly issues I would probably not to apologize for them but I would probably call these more like Linux issues or open SSL issues more than PyPy issues but certainly you do need to be aware of things like this that it is kind of a story to download up-to-date binaries on Linux unless you're gonna build them yourself which you can do which I'd recommend you do at least once but it's kind of annoying to automate a little bit because building PyPy takes about an hour or an hour and a half. Okay so provisioning there's also a story for deployment so in the normal world you're potentially used to doing deployment kind of like this certainly this is how we used to do it so when you do a deployment you push out the new version of your code and then restart the service. Unfortunately if you do this with PyPy your entire infrastructure is just gonna like get distracted doing jitting instead of actually doing any of the work that your service usually does because if you run let's say a web app with a whole bunch of processes and then you restart them something that we didn't really talk about before is basically the jit kind of needs to run and do the compilation when it notices that your program is hot in a certain area and it tries to compile a particular area of your program so if you kind of like deployed your whole infrastructure and then just restart all your services for the first X number of seconds your entire infrastructure is gonna basically be compiling stuff instead of serving any requests so that's not that great so instead what you have to do is kind of like a more like rolling sort of thing so what we do is fairly simple literally right now what we do is we basically go off to all of our services all of them kind of pre-fork so we actually use G-Unicorn to serve web apps let's say so we have G-Unicorn pre-forking 24 child processes and then instead of doing a restart which would cause the whole machine to start compiling instead of doing anything useful we basically kill off each of the processes one by one with a delay in between so that kind of like lets the machine keep serving some requests while still actually restarting the process it's not foolproof certainly and it complicates deployments a little bit because if you do change something that needed to actually change in the master process too you need to know that you need to do a full restart otherwise when you fork you won't actually get whatever change that you wanted to so for example if you actually change the underlying version of PyPy that you were using and then you fork from the master process that you didn't kill you're gonna get the wrong binary you're gonna be running the old version this is kind of a funny problem that actually happened to us unfortunately and this is now complete marketing so I apologize PyPy was too fast in the sense that when we switched our last remaining app over to it it actually blew out or kind of saturated our firewall so we literally had to roll it back I think it's probably still rolled back right yeah so it's waiting for firewall upgrades so beware of your network it's a great problem to have I just wanted to put GDBM on a slide just because how many people in this room know what GDBM is? Is it more than one? None How come you guys aren't raising your hand we use it? GDBM is basically the sequel light of key value stores yeah it's a DBM so how many people now know that it's in the standard library so yeah so there's a standard library module implementing GDBM we use it for okay reasons PyPy didn't have the GDBM module when we actually switched to PyPy three years ago for quite fantastic reasons basically because the CPython didn't have any tests for it so they didn't know it existed so like PyPy runs the CPython test suite but CPython had no tests for it so the PyPy guys didn't notice it existed so they had to add it afterwards so I put this up here mostly to tell that story but also because those sorts of things do come up every once in a while where unfortunately like every piece of software CPython's test suite has a bunch of holes in a couple of places most of them not often traveled like GDBM and so every once in a while you run into something that like doesn't work on PyPy but maybe doesn't really work on CPython either because there aren't any tests actually verifying that does the right thing so every once in a while this does come up where you'll kind of hit a dark corner of PyPython and it won't work because there's no tests verifying that it works upstream either and then I would be remiss to not talk a little bit about profiling not very much about specifics so like if you're in the CPython world I'm sure you're fairly used to running let's say the C profile module or like time it or sort of like these standard library little profiling tools maybe you're used to other ones too we have to be a little bit careful when you're using PyPy how you interpret their results so certainly because of JIT warmup like we talked about before if you just run some sort of profiling without doing that your results are pretty much meaningless because you're measuring the portion of your program your timing things in your program before they actually have sort of reached their final characteristics but once they do reach those final characteristics the sort of like information that you're getting out of C profile is still misleading because it's still telling you about kind of like Python code that's executing instead of giving you information about potentially Jitted code that's executing and so the world of profiling is a little bit different for PyPy there's this thing called VMProf which is somewhat immature but definitely usable it will tell you things like what portions of your code have been Jitted already it'll tell you like your code has been 80% Jitted and it'll tell you to sort of like which functions have been Jitted like there are sort of like internal PyPy tools for telling you what the JIT is doing when it's compiling all those sorts of things so there's kind of a slightly either expanded or modified tool set for doing profiling which you do need to be aware of and then before I get to this question which is obviously the most subjective one in the whole talk there are just slipped my mind so never mind we'll just get right to this so why doesn't everyone use this if I remember we'll come back to it so the first reason which probably covers 90% of the answer to this question is like performance doesn't matter for most programs performance doesn't matter specifically if you're IO bound like we said before so PyPy is gonna help you again kind of get rid of that step where you're executing lots of instructions instead of executing some smaller number for most applications performance doesn't matter it's fast enough I like mentioning because it makes it kind of like beefs us up as like Python people there's a guy, Greg Wilson who likes to talk about the study done by someone whose name unfortunately I've forgotten where basically a bunch of people were asked to implement a particular algorithm in Python, in C, in Java and if you haven't read that study the results basically said like the ultimate source code that was produced at least between let's say Python and C differed by like 1.3 times so whereas the languages have very different performance characteristics when you ask humans to write programs in them they tend to usually write programs that are roughly the same speed even without all the stuff that we've talked about so it's kind of an interesting side study which I will try linking to the slides so even if performance doesn't matter occasionally performance doesn't matter yet because you have those other solutions that we talked about before and then once we get past those two we get into kind of more interesting answers for the remaining 10% so if you're deeply invested in C Python again the scientific areas of the ecosystem do tend to have issues moving over most of those are related to how deeply those that ecosystem is already invested in external languages so we mentioned you know PyPy is going to make your Python code fast and it's trying to discourage you from taking sections and writing it in some other language unfortunately because that ecosystem already wanted fast code large portions of that ecosystem are written in other languages Fortran, C, blah, blah, blah and then this one so there is PyPy for Python 3 I've never heard of anyone running it in production but I don't run generally in circles where I hear about people running Python 3 in production regardless so if you are running Python 3 in production maybe try it out otherwise for us let's make code faster was an easy choice but certainly this does come up as a reason why people don't run it so this is my personal guess as to why more people don't use it is marketing that's why I'm here I don't work with them but like you should run it our internal policy which you guys should know is basically like all new applications target PyPy directly there's no reason why not to use it for everything at least for us and that's it questions?


What is going on guys welcome back to the channel and I do appreciate you being here today I bring you another destiny 2 video and today I bring you a simple guide into getting the Isenagi's burden Castle is done as fast as Possible, but before we go any further guys if you do enjoy the video and would like to show your support You can by hitting that like button if you are new around here and enjoy daily destiny 2 videos be sure to subscribe Okay, so the Isenagi's burden exotic sniper rifle The catalyst for this thing was introduced with the heroic Menagerie To get the catalyst for the Isenagi's you have to have a complete chalice of opulence Basically having everything unlocked and then you have to kill the last boss in the heroic Menagerie This drops the catalyst for the Isenagi's burden So what actually does this catalyst do and what change does it apply to the Isenagi's burden? While the whole shot the shot which you charge all four shots into one by holding that reload button It gives this shot 20% more damage against all enemies And you can see the difference is unscrewing now where I tested it on a last set to boss So won't you actually get this catalyst to drop from the heroic Menagerie to actually be able to apply this to the Isenagi's burden You have to get roughly 500 kills in PvE Now it might not be exactly that much, but it's around that sum I can confirm it's 100 kills within PvP though, and that would take you days in my opinion But the strat I bring you today is by far the quickest way in the game Now I know many people will be saying why don't you just do it in last sectors or load up the escalation protocol Or do it within the Leviathan? Well, they all would work, but they would take considerably more time because at the end of the day you need special ammo And it doesn't drop as much as you'd like and you spend most of your time hunting for that special ammo So yeah, I wouldn't even bother with them strats, definitely use this one Now make sure you have raid banners, these are obtained from Petruv on the Dreaming City It ain't a must, I mean they ain't a must, but they will help you get this done way quicker So once you have them, maybe 10 or 15, they definitely should be on the furrier You want to load up the last wish raid upon the Dreaming City And make your way to the Wall of Wishes You can follow this route on the screen now if you don't know your way there But finally here So once here, enter this code you see on screen now Once you've entered that, just stand on that center pad You will then be teleported to Shirochi, and this is where you get that work done Here if you don't have any special ammo, just drop a raid banner But before doing that, if you have any armors with sniper rifle benefits on Such as sniper rifle reserves, sniper rifle scavenger, or special ammo finder Make sure you use as many as these as possible These allow you to get easy special ammo without having to use a raid banner every single run So once you have that special ammo, just simply walk up to the door The door will open, and you'll be flooded by take and throw These take and throw are your food, consume them all Kill as many of these enemies as you can until you are either out of enemies or out of special ammo Then just simply kill yourself by letting her son count down The enemies kill you or jump off the map Then simply rinse and repeat until you get this done Now I managed to do the whole thing here, missing many shots and not really pushing it Went trying to do it as fast as possible, and I got this done in about 45 minutes So it shouldn't take you any longer than that in true reality And that's it guys, this in my opinion is the fastest way to get the catalyst done For the Izanagi's Bird and Exotic Sniper Rifle And on that note, we have come to the end of the video Guys, if this video helped you out leaving a like really helps me out If you're new around here and enjoy daily Destiny 2 videos, be sure to subscribe And if you never want to miss a video, I upload and return notifications And by hitting that bell button But guys, thanks as always for stopping by I hope you enjoyed the video, and hopefully I will see you on that next one


I'll catch you in the next video


Yay! Speaking of Python, we've got some Python on hardware stuff going on. Some good stuff this week. Yeah, some good stuff. Let's get stuff every week. Yeah, let's start off. We'll come back to kind of the lead story, which is side-labs introduced to support for CirclePython. That's going to be kind of our lead story. We just finished up the hack chat, mentioned that earlier. OpenSauce, if you want to look at some of the coverage, there was an event out in San Francisco that's like Maker Faire like. There was a lot of cool projects using Adafruit stuff and CirclePython. I think that it's, there's like the prop maker community. It's starting to like intersect a little bit, where they're because like it's like Adam Savage world, and there's like. Yeah, the YouTubey social maker. Yeah, there's definitely like, these are more for the, the audience was more of like the YouTubers that were like building props and things like, if you're a YouTuber, you're doing lots of things like, oh this movie just came out, we built a thing that's kind of like it. So, kind of neat. Check out some of the coverage that's around the web, but also MakerBulls and Wine, and you can check out some of the things that most of us all there. Lots of different projects. You can check out Todd's latest with some CirclePython synthiotricks. There is a MicroPython class for Arduino. We are merging support for this board on CirclePython as well. It's happening today. Yeah, the latest, one of the latest boards from Arduino on CirclePython support, but the big news, and you can check out all the different projects and more. The big news for the week was, PsyLabs now supports CirclePython, and that includes boards from SparkFun. Yeah, they're feather like board. Yeah, that's the big tent that we like to build. So PsyLabs now has CirclePython support for a bunch of their boards. We have, well, we put it all over our socials, but also it's in our press section. We'll play a video in a second, but Lady, why is this interesting? Why is this big deal? What does this mean? Why should people care? This is cool, because first off, this family, which we covered on INMPI a few weeks ago, so people want to see a more in-depth look into this chipset. It's a ZigBee, I believe, also Bluetooth, low energy chipset that can do matter and thread. And so if you want to do IoT projects and you want it to work with the newest system that will now allow anything to work with anything, so you can work with Google or Nest or Alexa or Siri or HomeKit, you don't have to worry about writing code for each platform if you're using matter. PsyLabs, the MG24 and other series chips do this, but one of the challenges with these chips is that they're so advanced. These are Cortex chips that have so much firmware that it's actually like compiling and uploading and doing the workflow can be a little challenging, which I think was CircuitPython is a great match for it. I think they haven't quite added wireless yet for CircuitPython, but they just released this, so I'm sure they'll add it later. But this is a totally new family of chips, like we've supported Dispressive, and Atmel Microchip, and Falcon 2040, but this is the first time we're supporting PsyLabs and they have a huge amount of marketplace, especially for wireless chipsets, so it's really great to see that there's another family. Also, for all of y'all out there, if you ever want to do anything with a PsyLabs chip and you didn't know or it was too intimidated, or it's like, okay, what am I going to do? If you do anything with CircuitPython, it actually works. So they made a video, we're going to play it. CircuitPython is now available for Silicon Labs XG24-based Low Power Wireless Development Kits. The low-cost Thing Plus MGM 240p board from Sparkfun provides an accessible starting point with the quick connector providing the option to add a wide range of external devices. The XG24 Explorer Kit from Silicon Labs is another entry-level board that features both quick and micro bus expansion options. Finally, Silicon Labs XG24 Dev Kit comes with a wide range of sensors already on the board, making it an excellent platform for rapid prototyping. Once again, the quick connector allows the easy addition of other devices. CircuitPython is a beginner-friendly programming language that simplifies coding for microcontrollers. To get started, visit the CircuitPython download page. Then search for PsyLabs to find and download the CircuitPython firmware for your board. To kickstart your development, you can clone the Silicon Labs CircuitPython applications repository on GitHub, where you'll find a growing number of example applications and instructions on how to get up and running. CircuitPython also comes with drivers for a wide range of third-party devices that can be easily connected to the XG24 boards. Visit the Silicon Labs CircuitPython application GitHub repository and start exploring the creative opportunities offered by CircuitPython for the EFR32 MG24 low-power wireless microcontroller from Silicon Labs.


So, I decided to check out an old XP farming spot and what I found people I just had to share with you. How's it going guys? My name is DPJ and today I bring you another BR3 video. If you do enjoy living a life, it really helps me out and subscribe if you do want to see more. So, the good old Scraptrap Prime XP farm was the best place to farm XP. Well guys, it's gotten even better. Because upon me going here to see what's up, I noticed something amazing about this spot. I'm not sure it was great for XP, but it's even better now because of the fact of the car tower enemies that spawn down here people at an insanely crazy rate. What this indeed means is this place right here outside of the Villa Ultra Violet is an incredible farm for those exclusive items found only from the sub-buses to Joey Ultra Violet as they spawn down here at a crazy number guys. Certain single runs at this place are seen three or four sub-buses per run spawn in. Three or four. Before with the chance of dropping, you'll add loot exclusive to them, which you'll see on screen now. What makes this even better than the Villa Ultra Violet farm I did post is the fact these sub-buses spawn in random each time. Farming the Villa Ultra Violet sub-buses are set to any random too and don't change unless you quit out and restart the process. None of that is needed here guys as they all randomly spawn in and what's great about the Scraptrap farm is the fact you don't have to quit out, you don't have to fast travel.  Wait a few seconds then run back, the fight will restart, rinse the repeat as you please people. Now the Legendaries down here are pretty good too as you get DLC 1 Legendaries from Scraptrap, non-DLC Legendaries from the other enemies and you also get to avenge the cartel exclusive Legendaries from the cartel mini-buses that constantly spawn in here too, as well as earning XP at the same time, so this is a great spot for Legendary loot and XP, so enjoy people. I just thought I'd make this video bringing this spot to your attention as I know for the most part many people have forgotten about this, like I initially did. I only went here to see what's up, so I enjoy it guys, but on that note we have come to the end. If you guys enjoyed it, leaving a like really helps out, if you're new and who want to see more Boilands be sure to subscribe and if you never want to miss a video workload you can turn notifications on by hitting that bell button. But guys thanks as always for stopping by, hopefully you enjoyed the video and hopefully I will see you on the next one.


just saw get out earlier this afternoon and boy do I have do I have mixed feelings about it all very intense the the short version is the first half of this film is one of the most perfectly crafted cinema experiences I've ever seen I use the word crafted very deliberately there because it was beat-for-beat moment-for-moment perfect one of the most perfect allegories for racism that I mean I've never imagined such a thing being so so perfectly made and getting its message across so perfectly I keep having to use the word perfect but the end of the film turned into Friday the 13th and then it ended on a joke a ha ha funny joke and I felt cheated so that's the short version for people who have not seen the film I still recommend it the things that the film has going for it are so strong that I think everybody should see it and I recommend it because of that now for a longer analysis like I said this movie every moment perfect I'm not all that familiar with Jordan Peele's work I know he's one of the members of the the comedy team key and Peele I've never seen their TV show but I know their reputation and every clip I've seen from their show I've come away thinking wow that was excellent satire that was really excellent satire well this movie is satire in the sense that it contains so much truth in an allegorical sense it boy it's really difficult to explain because of such a visual and emotional experience if you've seen the trailer and you should you if you're like me you'll come away from it thinking wow they they showed me too much in the movie just then well no they didn't there's a great deal more to the movie than is in the trailer so go ahead and watch the trailer and you'll get an idea of what kind of really severe deep cutting they're doing here to get across their point about racism this film has done an astonishingly beautiful job of illustrating how the the problem of institutionalized racism is not separate from personal racism that the the the little constant everyday examples of what we would call benign racism really are the root of the institutionalized racism and the the institution uses that as a way to strengthen itself and to keep alive and that the people we always talk about institutionalized racism as an institution and it is literally and in this film it wow it illustrates so well that the people who run the quote unquote institution they're not racist accidentally it's not just a symptom they're doing it deliberately they know what they're doing I was sitting there through the first half of this film with my eyes glued to the screen just in astonishment at how good this movie was how every performance not just the direction not just the editing but every performance of every person who appears in this film their performances are perfect exactly perfect I didn't see any moments in this film that I thought were out of place or clumsy or that I would have done different I was completely caught up in it how good this is you see that picture of the guy on the poster with that that look of despair of total despair and shock on his face and how good just that that one picture is on that poster just there are multiple performances like that in this film good Lord they're good so I was I was thinking as I was watching this film how this is great and I have no idea where it's gonna go I have no idea what's gonna happen and then it moved into a different area entirely and I was thinking okay it's keeping up with the the allegory with with the storytelling representing actual real world problems this is moving into an area of the rich versus not only minorities but the rest of us and the problems of the older generation stealing from the younger generations and I was thinking wow this is this is getting bigger this is getting bigger and I still don't know where it's going and like I said at the beginning it turned into just a horror movie and but it still could have been saved except for right at the very end I mean the very last moment of the film there was a joke the the goofy comic really friend there was a haha joke and I sat back and I was like really really so so this although I'm still astonished at how good this film is and and how how much how strongly I recommend that everyone see it and talk about what they've seen seriously talk about what you've seen in this film it's unfortunately it's another example of a creator whether it's a book author or a filmmaker making a promise with the beginning of the the project and then dropping the ball but this now see I could be wrong I could be this movie was so perfectly crafted every second of this film was so meticulously made that I have to wonder did did Jordan Peel here did he do this on purpose and I'm wondering if he did it on purpose what was his purpose because what happened to me was in the first half of the film I was completely caught up in the horror of institutionalized racism I was I was terrified for this guy and I was seeing and the fact that this movie was released today released now with the things that are happening in the USA and in the rest of the world consequently today I was seeing allegories for all the rest of us the title get out it's really meaningful and and I was really caught up in those thoughts and in the the the messages that that he was clearly shooting for and then thinking about them as the film was going on but by the end of the film I had forgotten all that and I was caught up in the chase and the escape and then at the end there was that oh haha he got away everything's fine haha and after my initial disappointment as I was thinking about it on the bus ride home I was thinking you know what maybe that's exactly what he manipulated me into maybe he understood that that's how the movie would be accepted by audiences that the film was so well made I have to wonder did Jordan Peele do that on purpose I'm gonna wonder about that until I hear him say it