Fantastic. All right, we fixed it. Good job team. That's very weird. All right, okay, if you couldn't hear anything I'm saying, then I'll do a quick recap. Okay, so the basic idea of this library is that we're going to spin up a bunch of EC2 instances. And on those EC2 instances, we're going to run some short-lived jobs. So this is the example that we wrote last time. You can declare a bunch of servers. And those servers are our library is going to take care of spinning up those machines. And then it's going to give you an SSH, a way to SSH into all those machines and run whatever commands you want. After all the machines have been set up, you get to run this closure at the bottom. And in that closure, you have SSH connections and the IP addresses of all the machines that were spun up in the end. And if you recall from last time, we have this severe limitation that basically only I can use these machines, because they're launched with a security group that just allows SSH from my machine. And they only allow my SSH key pair. And that's really inconvenient. And so what we're going to try to fix is this particular problem before just to give us a way to get back into the code. So we're going to start with something that's arguably fairly straightforward, but it might be instructive, and it might be useful just to walk through that and re-familiarize ourselves with the whole Risotto EC2 setup things. In particular, we're going to have to create a security group, which is how on Amazon you configure the firewall, essentially. So we're going to set up a security group that allows all of the machines that we spin up to talk to one another, and also that allows our machine to SSH to the machines that we spin up. And if you remember from last time, most of the Risotto EC2 API methods are basically they take a request thing and they return a result thing. And so we're going to be using that. We'll first create the security group and then we'll add the rules to it later. And the second thing we'll have to do is set up the SSH key pairs. In particular, what we'll do is actually use the fact that EC2 has this feature where you can tell it generate a key pair for me and add it to this machine. And then we can use that and import that into our SSH library, the one that we're using, so that in theory we can connect without the user having to give us names and passwords and those kind of things. So what we did here was we have to generate a name for our security group, which is fine. We also need to find the machine that we're on public IP address so that when we try to connect to this remote machine, it will allow SSH connections from us. Now, it is possible also to just allow SSH from everywhere. And maybe that's what we want to do. But for now, we're just going to take actually let's do that. That way we don't have to check for our own IP. Yeah, that's fine. Okay, so we have a group name. And that means that we're now going to have to call this method over here, which takes a security create security group request, which is one of these guys. So if you recall from last time, basically the way you interact with this API is all of the, all of the types that describe requests implement default. And then you override only the values you care about. So in our case, we're gonna have a request, which is going to be a risotto, EC2, one of those defaults. And in our case, we want to set the group name to be as an option. Right. So to be the so we randomly generated a group name here. In our case, we put a prefix and then we use the Rand crate to generate some rasky some random ASCII characters. So now we have a group name. And then we're just gonna, whereas the right, so we're going to do basically the same thing as we did last time. So last time we integrated with the failure crate. And this lets us easily propagate errors up to our callers. And we want to keep doing that same thing, we don't want to break the failure, we basically don't want to unwrap anywhere, anywhere in this library. So we'll go here, copy that code. In our case, it's not request bot instance, it's create security. And we want to map this into a failure error. And one of the cool things about the failure crate is that you can add context to errors that you propagate. And so we'll take advantage of that here. In this case, we failed to create security groups for new machines. And it's okay for us to just return here if we get an error, we don't actually need to do like, further down, we do a bunch of retrying to carefully undo what we did or to try to progress. Whereas in this case, this is sufficiently early on that it's fine for us to just return, because we haven't left any machines running if this call fails. So we don't even need this format. Let me just do this. So that's the context we want to provide. And now once this finished, the result that we get back here has a group ID. And that group ID, we're going to have to refer to later whenever we want to modify the security group. In our particular instance, what we want is we want that security group to also contain the the rules for allowing us to SSH to the machine, and also for the machines that we spin up to talk to one another. So in this case, group ID is going to be that dot here. Notice how the API returns an option group ID. In reality, it's this is basically because the risotto crate is somewhat auto generated. And all the Amazon API's issue, or give JSON requested responses where you never know whether something is actually a value, the Amazon API guarantees that this will not be done. But the library does not actually provide this guarantee. So here we're okay with giving expect, which is like unwrap but with a reason. So we will say AWS created security group with no group. All right. So now that we have a security group, we now need to add rules to that security group. So think of this as we sort of created a firewall. And now we need to set what rules we want to add for that firewall. So this where update security rule. Yeah, so if you look at the one that I created previously, you can set inbound and outbound rules. And in particular, what we want here is inbound rules. We want to say that any external machine can connect to this machine. And so we want where you go here. So we want to update security group security group rule descriptions ingress. Wow, what a mouthful. No. Okay, so here, we basically want to add new rules, we're going to use this one. And notice how the pattern here is the same. You give it a request and the name is clearly auto generated based on the API. And you get back a result that's the same thing as the request, or similarly typed as the request. Okay, so we're going to make one of these. It looks like a bunch of pain, but all right. So the pattern here is the same, risotto, easy to this, and then a default one of those. And then for that, we're going to set the group ID is going to be the group ID that we got back from our create security group. The group, what's group name? No, we don't need that. Because we have the group IDs, we don't actually need the group name. And then IP permissions is the one that's particularly interesting to us. This is going to be a VEC. So that's great. And it's going to be a VEC of what IP permissions. Yeah, this seems great. Alright, so we need to think now what IP permissions we want. Well, we're going to want this implement default as well. Great. So there are basically two rules we want to add. We want sort of cross talk. So this is the machines that we spin up, we want to be able to talk to one another. And we want access, which is going to be our SSH access to that machine, right? So access is going to be things here, we don't want to set a from port, we do want to set an IP protocol, it's going to be what TCP? Yes, this is actually sorry, what kind of autocomplete do you run for rust? So I use RLS. And so this is NeoVim, and I'm using language client. So this is an extension for NeoVim and VIM, I believe, that integrates well with RLS. I also use Racer. This is basically sort of the recommended VIM setup for rust. RLS is also now distributed with with rust using the rust up tool. So if you look at rust, where's it? Yeah. So there's a component called RLS preview that you can add to your rust up and then you get all most of the autocomplete just starts working. It's pretty great. One thing to point out here is this is something you're going to run across in rust code all the time is you have a static string, so an SDR, and you need a string. There are so many ways to do this. There's two string, there's two owned, there is string from, there is format, and all sorts of things. I personally like two string. I wish there was a shorter way to type this, there's been some discussion among the rust team of having STRs automatically can be sort of converted into a non-mutable string with a capital S, but that's some way off. Okay, so we want TCP, this is going to be SSH access. So what we really care about is two port. We're going to set access dot two port to be, this is take an I64, great. So two port 22. And is there anything else we need to set? We need to set one or more IPv4 ranges. Okay, so set this, it's going to be a this is one of the slightly unfortunate things about using an auto generated library is that we end up with things like this, which just should not be necessary. I've chatted a little bit to the people who wrote this library after the last video we did, and I think they're aware of some of these problems. Okay, so we need an IP range. This one, we're just going to do this. Ooh, can I do this? Okay, I can do this. I think that's the syntax for it. Specifically, I want cider IP to be some, I want access from any IP address. Yeah, so this syntax is really nice. It's saying, I want to set this field, and all the other fields set to default, we could have done that here for access to, like arguably we should have. Ah, it's fine. Okay, so for, so that's the SSH rule we want. So we know that that's one of the permissions we'll want to set. Cross talk is a little interesting. So by default, Amazon also does not let your machines talk to one another. And this of course is something we want. So we want to set that up. In our case, it's going to be fairly similar. So here, except cross talk, except we don't want to set a two port, we want to allow every port, and the IP range is going to be, we don't actually want every port to be public that would make us and everyone else really sad. But if we look at the API, it's like this range, which is the default virtual private cloud that Amazon gives you. So we're going to give access to that by default. And now we have cross talk. Okay, so in theory, this is going to set up a security group where the where we can SSH the machine and the machines can talk to one another. Right. And now this to do and this becomes a good question. Does it only take a string? That's a little bit sad. Um, probably not. So this is request spot launch, request spot launch, that one. This takes a security group IDs instead, which is what we really want. There takes a vec of now we're almost there. Uh, I guess this is going to have to be, um, all right. So in theory, that should just like have the machines now automatically be able to talk to each other. Oh, we need to call the API. We're going to do the same thing here. So if it errors out, we're just going to give up. There's some cleanup that we may want to do here, like delete the security group after we're done. But I'm going to ignore that for now because that seems a little bit frustrating. All right. So this one here failed to fill in security group. All right, let's like just see that this works. So we're going to run cargo run example. Just to see what happens. Oh, I did not like that. Update security group rule descriptions ingress. What a name for a type though. What's it called? This thing. This is the terminal theme. Yeah. So this is, um, base 16 atelier dune. So base 16 is a great, um, it's a, like a framework for building themes. And specifically, so it like all the themes are built with this work with all sorts of backends. And specifically, I'm using the theme that's called atelier. And this one, it's pretty nice. Anyway, um, that's a little disappointing. Cargo dog. Anyway, okay. So in theory, this is going to set up access to all the machines. We'll see what this documentation gives us after a while. The other thing we'll have to do, and this is where this is going to be a little bit more interesting because now we're going to have to generate an SSH private public key pair, put it in Amazon and then use that same key in the SSH library we're using. So we sort of need these two libraries to talk to one another. So while that's compiling, I believe this function is getting pretty large now. So we should probably start splitting it out into functions. Set up network firewall machines and then construct key pair for right. So how are we going to do this? If I recall correctly, there is a generate. So there's import key pair, but there's a create key pair, that one. Yeah, so this one, easy to generate the key pair for you. So we don't have to deal with that particular part. And then gives us the private key and well constructs a public key with some name. So in our particular case, again, this is the same sort of pattern. We're going to issue a create key pair request, and we're going to get back a key pair. So here, we're going to say, is it going to be a risotto EC2, one of these defaults. And let's see what this request is like. Oh, that's moving slowly. I think my don't have enough course. Yeah, so here, there's very little we have to give, we just need to give a key name. I think again, what we're going to do here is the same thing as we did for the group name. I'm going to say that the key name is going to have some random name, we don't particularly care what it is. And we're going to do the same thing again, as we did before of doing create key pair fail to generate like so. So in theory, now Amazon gives us a key pair. Again, if this fails, we might as well give up because we haven't spawned any machine yet. Here we go. So what does this say that risotto EC2 gives us EC2? Sorry, so this was cargo doc dash dash open, which is a super convenient command because it compiles the documentations for all your dependencies as well. And then you noticed but our crate does not have any meaningful documentation yet. This is one of the things we're going to fix. But it also all of your dependent crates are listed under crates here. So in this case, we can go to risotto EC2, we can go to the EC2 trait. And now Oh, that's interesting. Does not have the same methods. Oh, you know what? This is totally this documentation here is probably generated for the master branch of risotto, which is not the one we're using. Let's see if there's another there's revoke security grouping address. Authorize security grouping address. Alright, I guess we'll do that instead. Should not matter terribly. My guess is the setup is about the same. Hello there. Oh, welcome. You were here last time. Okay, so we're gonna authorize we're gonna use this authorizing instead. My guess is the API basically exactly the same. Pretty good. This takes one of these now. And look at that. It's basically the same thing. In fact, it's a little bit less convenient because this one, wait, sorry, you can't specify this parameter when specifying a source security group. That's interesting. That's very weird. I don't know why this parameter is here. Let's just see what happens if we don't set it. Okay, so we're gonna create a we're gonna authorize ingress to port 22 on that one. We're gonna authorize. I guess actually we can just use this one directly. We don't actually need to set up multiple permissions. So this is probably fine. So IP protocol TCP. And then we're gonna have cider IP. It's gonna be just this thing. Right? Yeah, let's do that. Seems easier. This is gonna be cider IP is gonna be that I see why I wanted to do that. Okay, so these are IP permissions. So we've set up two of those now. So we do actually need them to be of this form, right? Let's see. So it takes can be used as my multiple rules in a single command. Yeah, so this is again one of the problems when a library is auto generated that there's some conflict here because all of these fields like cider IP from port to port are also on IP permission, which you can use to set multiple of them. So it's a little bit unclear how we're supposed to do this. I think it's okay for us to just set IP permissions. So we set the group, the group ID and IP permissions and we don't set any of the others. So they're just going to be set to none. All right. Let's see just whether that compiles at all. That looks about right. Expected option found back. So this probably has to be some group ID not found. Oh, right. So down here, we direct dot key name is going to be right. So this is we're going to generate a key pair. And then let's now see. Okay, great. So I'm just going to comment out this for now. And then just see now that this builds, just see whether the value moved here. Where did I move it? 137 previous iteration of the loop. It's not super ergonomic, but okay. No, I just want to I want to run this just to see that it does the right thing. Okay, so if you recall from last time, the example that we wrote this thing worked fine. And now we haven't really changed anything in the sense that all the other code should still work the way it is. So all we're really checking is whether the security group setup security group setup works correctly. And if it does, then everything should basically not change. So we're going to run that. And then we're going to go over here. We're going to close this. And we're going to look for spot requests. So in theory, this should now spawn up for EC two machines. One, that's going to print the hostname. Three, they're going to print the date. And then it's going to print the IP addresses of all those machines. If everything went according to plan, it did not. Oh, that's so Amazon, whenever you interact with the Amazon API, you need to set this like AWS secret key. They'll just create security missing parameter, the request must contain the parameter group description. I see. So they're really lying to us when they say this is optional. We will set description is what are we going to call it, temporary access group for tsunami VMs. Let's see. Valid value must specify both from and to ports with TCP and UDP. Really? That sounds weird. That sounds like a lie. Here. What did we set as there's no source port here? So why is it saying that I must set a source port? Let's see. Port rain. Oh, it's the start and end of the range. Okay, so we want from port 22 to port 22. And down here, we want from port. What's the actual range? Like zero to 65, 535. 65, 535. The console magically provides. Oh, yeah, yeah, that's a good catch. That's what I figured, probably. It was more I thought from port was the source port, which of course is clearly not the case. Must specify. Wait, what? It's still telling me I must specify that, but I am. Oh, this is totally this thing we saw coming back to bite us. Remember how? Yeah, so this thing has both this IP permissions and the fields directly here. So it seems a little bit as though we're required. Hmm. Right, let's just do this, the straightforward way of doing two requests. It's a little bit sad, but we can do it. So we're going to do instead of having this be access, we're going to do, we're just going to do this directly on this. So instead of using this IP permissions, which seems to be causing us all sorts of problems, just going to set this directly. So, and then we're going to do a request. And then we're going to do a second request right after. Luckily, we can reuse many of the fields from the previous one. So for a crosstalk, this so this top thing is going to set up SSH access. And the bottom one is going to set up communication between VMs. Hopefully, at least, we're about to find out. Do we think it's going to be nice to us now? It's looking promising. See what happens? First of all, is there a new security group? There is one right there. Doesn't have any rules though. This one does. Okay, so that has, that looks like the right rules that we want. SSH from anywhere and TCP in, spot requests anywhere, spot requests, canceled. Are they finished? No. There's not a spot request yet. What about instances? Oh, that's right, because it, it starts the spot request, waits for the instances to come up and then stops the spot request. Otherwise, Amazon is going to spawn just more machines for us. Great. So some machines are running. That seems pretty promising. So now it's probably just waiting for them to boot. So really, now we're, what we're about to see is whether or not the SSH access work, like whether the code we set up to actually allow SSH access work correctly, but it looks as though it does. What we can check is whether it's attached to the right security group. It is, right? Yeah. Okay. Yeah, that looks like SSH. Notice how, so this is the second thing we're going to fix currently is prompting me for my SSH key, because it's logging in using my SSH key, which is a little bit unfortunate. So the next thing we're going to do is we're going to have the library pick an SSH key for you and automatically do all the integration. And that's going to make this a lot more pleasant to work with. It's also going to mean that it works for other people than just me. Okay. So we got one host name and one date print. In theory, we should get two more date prints. And then we should get all the private IPs suggests that the machines are taking a little bit longer to start up. While that's running, I guess we can go back to this. So constructing a key pair. Oh, actually, don't want the response value for this, we don't care about the results. As long as it's okay, it's fine. Oh, so underscore is a way to say, just drop this value immediately. Don't bind it to any, any variable. I believe it also drops the value immediately. It doesn't just sort of bind it to some hidden variable and drop it at the end of the scope, like it does with normal variables, it actually drops it immediately. So that's something to be aware of. But in this case, we don't particularly care. So this is going to generate a key pair for us. And if we look at finish yet, still running, it's a little bit disconcerting. Can I like SSH to this machine? Can I SSH to this machine? That's a little weird. Okay, so the library is lying to us. It's stuck somewhere. Well, this is a good time as any to look at some gdb debugging. All right, so we're going to attach to the process that's currently running this one over here. And we're going to look where it's. This suggests to me that oh, okay, so when we connect to something on SSH, if we get really unlucky, the connect is going to connect right before the VM is spun up so that it tries to connect to some other, the old binding for that DNS name. And then it's just going to hang for a really long time. There we go. Yeah. So now it went through. So this is basically the connect timed out at some point. You can adjust the connect timeout for TCP stream. We're not going to bother changing this here, I think. But see, so it ran the run thing. So it ran one server, three clients, and then printed all the IP addresses. So great. This means that our security stuff is working. Now we just need to get the key stuff to work. And then we can start moving on to rayon probably actually, and do parallel setup should be pretty cool. All right, so we're constructing a key pair, tearing down the machines, great. And notice how because we haven't written this code yet, the old security groups are just going to stay behind. Ideally, we want to write some code that cleans this up, but that's not terribly important for right now. So let's see. So the create key pair. The key pair request just has a name. So that's not terribly important. But the response, and this is where it gets interesting, has key fingerprint, key material and key name. The key fingerprint, we're not terribly, we don't care terribly about because it's essentially you're probably, you're a digest of your private key. But the key material is the actual private key that we're going to use to SSH in and the SSH library, if we look through this has a session. So for session, you can use her off with a pub key file. And this takes a private key as a path. It's a little bit unfortunate, because we're going to have the key in memory. So I actually filed an issue for this a while back. So specifically, currently, we're going to have to put this key in a file, but it's fine. That's true, we could use the IP addresses instead of the, instead of the DNS that might tell. In fact, is the SSH or we're currently using let me double check this. So the thing that creates the SSH connections is down here. So it waits until they're all up. And then once they're all active, it connects to, oh, yeah, you're right, public DNS. Just make that public IP, shall we? Oh, yeah, Rayon looks super cool. I agree. I've used it a little bit in the past. In this particular instance, it's a little bit weird to use Rayon because we're not really doing compute in parallel. We're just using it as a way to do many things in parallel, even if they're not compute heavy. But it still works really well for that purpose. Okay, so we have a key pair now. So specifically in rest dot, where is it? Key material. It's an unencrypted, PEM encoded RSA private key. And let's see, if we're really lucky, this takes a PEM encoded private key file stored on disk. Great. So we have the same encoding. So in theory, this should just work out in practice. Who knows. Notice also that this seems to require a public key, but it's an option. Is that an option? So this is the underlying SSH. So the SSH to library is just an FFI wrapper around a C library. Some user space threading is best in that case, right? Tokyo. So we could use Tokyo or something like that for doing the parallel setup. It gets a little bit weird. It turns out it's actually easier to do it with Rayon in our particular case, because the SSH library is not asynchronous. Specifically, the, this entire SSH library has a fully blocking interface. And so it would be kind of tough to sort of thread this on top of an event based library, such as Tokyo. If this supported asynchronous read, write of sessions that we could totally do this using Tokyo, I agree.  I don't know if there's a, there's a plan to write an async version of this, but I think that could be really nice. In fact, maybe that's something we will do. Who knows? Yeah, so we're gonna be calling this method, but we want to, we want to figure out why there's a public key here and whether we really want to give it, because the key pair we get back does not actually have the public key. And we can, there are ways you can generate the public key from the private key, but I don't really want to do that. Risotto async is coming soon. Risotto async will actually not help us that much in this case. Like it'll be nice in the sense that some of our setup code could be async, but it would not fix the issue of SSH being, not being async, right?  All right, so wait, why is, that's weird. Okay, so what we're gonna do is we're gonna look at the documentation for the underlying C library that this is using for the user auth pubkey file. So lib SSH2 user auth public key from file calls x public key. If lib SSH2 is built against open SSL, this option can be set to null. I'm gonna just assume that we're using open SSL and then set it to none, because it makes things easier. This is something that we might want to fix later on. You can generate the public key from the private key, a key file. All right, so we're gonna have a private key is going to be the key material.expect AWS did not generate the key material or new key. Again, this is something where the AP normally we wouldn't want to use unwrap or expect. In this case, the API says that it will give us key material. The key material key of the response will be set. Otherwise, the request would have failed in which case, which we would handle over here. And so this suggests that it is actually okay to expect here. If this fails, that means the API did not follow its spec. So like, we could return an error instead. It's a little unclear actually what the right interface is, and if you have thoughts, then feel free to raise them. But in this particular case, the API says that this should never be none. Okay, so now, when we launch, the key name we're going to use is going to be the name we gave, the random name that we gave to the key. So now this to do also goes away. This little VPC that we might do later, but we're going to ignore that for now. And now when we do the SSH connection, this is where things get really tricky. This thing is going to have to take in a path to the privacy. So in this case, a path is one of these dark theme. So it's going to take that. And then normally, we used to authenticate using the, you shouldn't need all those map error, there's a trait result x, which lets you call context on a result. That's nice. In failure, you mean, right? This one. Oh, I want that. That seems great. Let's do that. So here, we're going to use this trait so that a result gets a context. And then we can do this. And this. And this. Oh, I see you're also saying that I don't even need the map error. Because the result has this directly. So we can do even better than this, right? We can just do this. Right? That's the claim. Nice. Okay, so this goes away. This goes away. This goes away. Oh, that's so nice. Thanks, I was not aware of that. So let's see if that works. 70. This is a little bit annoying, but so does not implement for failure. Why does that not work? Interesting. Wait, machine does not have public IP. This is something else that we should fix. So our machines, we want them to have the public IP rest. We'll set that later. What am I missing for this error here? So I think we can do this. It should be necessary, but it looks like it maybe is no context found for that, right? So I should be allowed to do that, which is going to generate a an error which I can call context on and I should be able to do into on that. But for whatever reason, it doesn't let me. Okay, great. Yeah, so we want the public IP address here. We know that there's a public IP address. This is a suggestion to use IP and public IP instead of DNS to avoid the race. And what are we missing? Okay, great. So we're back to where we were. So now the place where we do, where we establish SSH connections now has a path to the private key or takes a private path to the private key. And then instead of using the SSH agent of the user, we're going to authenticate just directly using that private key file. So we're going to use, where's our SSH session? We're going to use this method instead. So we're going to do this. And in our case, that we want the username, we don't need to use a public key because we're assuming, oh yeah, I'll commit slightly later. I just want to finish this SSH feature because then we're going to move on to rayon and before that I'm going to do commits. And I will split them into separate commits, don't worry. Right, so we don't give a public key. The private key is the key path that we're given and there is no passphrase. And then we can do this here as well. Yay, this is no longer to do. This is no longer like that.  All right, I need the same need to use the trait to have it apply. Right, so now in lib 285. So now when we connect, in addition to giving the machine that we're going to connect to, we also need to give a path to the key file. This is going to be a little bit annoying because the private key here, again, we have in memory, whereas the library we're using actually requires us to have a path. So we're going to use a nice thing that, oh, I guess we should probably not rely on nightly, should we? There's a really nice write this one that just dumps an entire byte stream into a file, but it's currently nightly. So I guess we will do a file create and write all. So we're going to write key pair to disk. And now we're going to need a temporary file. Is there is there one in FS? I feel like there should be, but maybe there isn't. Is there a temporary? Actually, that's a good question. Do any of you know of a rust thing to generate temporary filenames or even temporary files? Can copy the source of that function locally? I mean, yeah, you're right. I could do that. It's not particularly long. It's basically just file create write all like it's a very straightforward function. Yeah, I think I think you're right that it's called temp file. Yeah, securely. Oh, hey, I know that guy. All right. So we're going to add to our cargo to Tamil. I'm going to have temp file. Let's see what the docs are like for this. Yep, that seems like just about what we want. So we're going to use this technically, like so. So what we're doing now is we're creating a temporary file that we're going to stuff the private key into so that we have a path that we can then give to RSA library. In our case, did that type check? Yes, great. So we're going to do temp file context with question mark fail to create temporary file for key pair. So that's going to give us a file. Actually, we don't even need the file. We can just do write all directly on this. I'm going to need IO for that, though, probably. So we're going to do write all of the private key dot context. In this case, the context here is could not write private key to let's see how that works. A string is not bytes, but it is now your editor is pretty low contrast for half dimmed phone screen. Let's see if I can do something about that. Is that better? Would that make you happier if it looked like this? Because I'm happy to switch it to this, actually. I don't know that it's really better because the contrast is still a little bit annoying. In general, it's just like more readable. I guess I can show you the MRC. Is this easier to read? I feel like this is just harder to read. Actually, how about this? How about we go with what's the high contrast one? I want to see what themes there are. Let's see if there's one still poor contrast with the colors. Yes, exactly. So doesn't he have a, that's unhelpful. Just want to see examples of themes and then we can pick one that has high contrast. That seems fine. Online editor for preview, maybe. Yeah, I mean, I think the contrast is fine if you're not on a dimmed phone screen. So I'm going to stick to this. There we go. All right. So the, yeah, that's a good point. It's not worth the sidetrack. All right. So the issue we had was that the string is not bytes. I just love the compiler. So it tells you that you can use as bytes, which is totally true. The other thing is that if you look at the Rust string, Rust string is string specifically, implements D ref into SDR and SDR implements D ref into, where's the D ref? S ref of U8. Unfortunately, the compiler isn't smart enough to do this double D ref for us. So what we can do is this, this now turns the, encourages the compiler to D ref the string into an SDR, which it can then use ASRF on to make into bytes. Do I actually need to D ref it? Fine. All right. I feel like I've had that before, but apparently not. Okay. So we'll do us a compiler says and we'll use as bytes. So now this in theory will write the private key into this temporary file. Oh, we do need the file name dot write all. And then we're going to actually do this private key is going to be now a path. And we specifically want to so the reason we want to drop the file is to flush it to disk. And so what we want is on the file, the temp file gives us back. That's right, we do want to name temp file. Does it not? Oh, that's right. Because the temp file deletes it because it unlinks it immediately. Yeah, I do think we need a name temp file. And this is because we need to pass a path to the private key into our SSH library. So we're going to use a temp name file, one of these. That's fine.  It's a little bit sad. But in this case, the key pair is fairly short lived anyway. So the problem here is the private key file is going to end up as on temporary directory somewhere. And that's not great. We could have it be surely we could have it been the current directory. That's not terrible. Okay, so the private key is going to be this one. We're writing all the private key bytes into it. And then when we later try to establish an SSH connection, we're going to now pass in the private key file path. So now I guess I don't need that function anymore. Yeah, so that builds. So now SSH in theory gets a path to the private key file, it connects using the private key file, which means that now it should never touch my SSH key anymore. So let's see if that works. So notice now that the code no longer has this it no longer hard codes, my key or my security group, which means that in theory now it should work on other people's machines. So let's see what this does. In theory, we should see a new key pair appear. We also currently don't tidy up the the key pair. So in theory, we should not the file the file is deleted automatically by temp file. But the the key pair that we generate in AWS, we should also clean up. Okay, that generated a tsunami key. And we should have a new security group. And we should have some spot requests that are immediately canceled a few seconds ago. Great. So we should have instances that are tied to the new key name that we made. This one, for example, is connected to that key, which also means that if I tried SSH to this machine now, I should not have access with my with my regular SSH key. Right. This should not work. Great. What if I give the one that it made in here? Uh, that's a good question. No, no, which file that would be this one, maybe? Oh, it's not terribly important. Let's see. So now, once these come up, if this now works, then that means that we're sort of done with the whole fixing this. So it works for other people. We could do a documentation pass, but I think the API is going to change a little bit when we do rayon. And so I want to not make that change quite yet. So while that now runs, let me make some commits.  I think I'm going to deal with the changes to the way we did errors first. Um, suggested here elsewhere. So none of this, maybe writing the SSH files could help in case they need to debug something. Yeah. So, um, I do sort of like them not being in the file system, but you are right that it could be nice to leave the private key files around. That said, if you wanted to debug this, you can also, um, to a known path. Yeah, the problem there is you have collisions would have to be a known directory. Um, it's also, you can always add more keys afterwards in AWS. So if you really wanted to debug a machine, then that's one way you could do it. Okay, so these, this, this, this, not that sort of this, that's gonna be a little bit annoying. So I want this. It's going to come back to bite me, isn't it? Definitely gonna get this wrong. Um, don't do this at home kids. Like so this goes here. Just ensure correct permissions. Oh yeah. Yeah. Yeah. So I totally agree that we could write this private key to like a well known directory somewhere that users could use later. That seems fine. Okay, so this, yes to that. We started using this trick here as well. This stays here. This. All right. So now the things that are left should, there's one more. So now all the, all the map error things that we fixed, we're going to commit them once. Use. Failure. Right. This finish. Yeah, did the right thing. Okay. So this suggests that our, the interaction with our SSH library is now working correctly. It is not using my SSH key anymore. It's setting up the firewall rules on its own. We're going to want to do a cleanup of this, which I guess we can do down here. That seems fine. So notice how at the end we terminated all the requests, right? And we also down here want to clean up security groups and keys. So we're going to do, we're going to do this very straightforwardly security group, which my guess is this is very straightforward. It's going to take a delete security group requests. And my guess is this just takes a group ID. Great. And we already have the group ID. It is group ID. And then we're going to not even retry. I'm just going to do ec2 dot delete security group. We're going to have the context of fail to delete security group. I guess clean up security groups. So then we're going to do the same for the key. This is probably going to be, I guess, delete key pair. Delete key pair. Yeah. So this is going to be the same. We're going to have a delete key pair request. The key name is going to be the key name, right? Yeah. And delete key pair is going to clean up key pair. All right. And that should type check is my guess. No. See, this is one of the this is another case where we see that API ergonomics are important. This one does not take an option. Whereas that one does. Seems kind of silly. Okay, great. So we are now going to commit some of the other things we did. Let's see. None of that. This is a separate change. We really changed this one. Yeah, I know manually, manually editing defiles is not generally recommended, but we're going to do it anyway. Okay, so this was switch to using public IP to avoid DNS race that was suggested by someone. And now the rest of this is security group. You set up custom security group and key pair. Great. I'll push that. Fantastic. All right. Let's do rayon, shall we? Okay, so rayon service definition files is worth them. So I don't know that this is about the service definition files as much as it is. There needs to be a library that I don't think it's feasible to just auto generate an entire library from the service description files. I think you get decently far for low level bindings. But I think you want a higher level API. I'm not really complaining about Rosado. It's super useful. And this this crate would not be possible without it. But I do think that there's a place for a library that sort of wraps just the base API in a sense that's sort of what we're doing. But we're doing it for a very targeted use case. Okay, so let's see. So the reason we wanted rayon, oh, I want scope guard to deal with that later. So rayon lets you do things in parallel. It's sort of the basic idea of it. In particular, if you have say an iterator, then you can, it sort of spawns a thread pool.  And it spawns as many threads as say, the number of cores on your machine. And this means that it can do a bunch of things in parallel. Normally, this is used for paralyzing compute, like if you have a vector and you want to do some compute on everything in the vector, you would use something like the pair iter that they have, which is parallel iteration over a vector to iterate over all the things do something with them like a map. But then those maps will be run in different threads and many concurrently. In our case, we're going to use this for setting up the the machines. So in our case, we have a list of the machines that we want to iterate over, and we want to run all their setup methods in parallel. So in our case, there are a couple of different ways we can do this. So remember how we have machines of different types, right? So we have the server type and the client type. One sort of stepping stone we could do is to within the set run all the setups in parallel, but not run in parallel across the sets. That would be a slightly easier. In our case, this would correspond to taking the inner loop of making a parallel, or we take the outer loop and try to make that parallel, which is going to require a little bit more work. I think we will that's a good question. Let's do the inner one first. Okay, so in general, there are lots of ways to use rayon. Para iter is a very useful way to do this. So if you look at the where is it? So here we want parallel iterators. Ah, link does not work. No, that's terrible. Yeah, so this is sort of an example of the kind of things you could do. In our case, we want para iter mute, which you can only do on certain types of collections. So that we get a mutable reference to each of the machines. In our particular case, the reason for this is because we're establishing an SSH connection to each one. And we want to store that SSH connection after we finish setup. So in theory, this should be as easy as iter mute to give us a mutable iterator dot para iter mute dot for each machine. And then do we think this will just work immediately? No, great. That's because I have not imported rayon yet. All right, rayon version 1.0.0, right? Yeah. Now we want extreme plate rayon. And we want to use rayon iter. Where's para iter mute again? So like a prelude in rayon, do you remember? I don't think so. I think I just want parallel iterator. Let's see. Where is para iter mute defined? Is it like index parallel iterator or something? I guess we can do this. Para iter mute. Into parallel, that's a terrible trait. This one, I guess I am using that one. Is it so this one is implemented for a parallel version of iter mute, and it's defined for anything that implements into parallel iterator. So in theory, I should be able to do this, right? There's a rayon prelude input import need to be used. So I don't think rayon has a prelude. That was the observation. Unless I'm missing something, I don't think it, oh, yeah, you're right. Clearly I'm blind today. Prelude. Great. That worked. Yeah. So now we're iterating over all of the machines in each set in parallel. And then for each one, we're establishing an SSH session, calling the context, and then setting the session. Now notice, one thing that gets a little bit more complicated here is that now the errors that occur within here are sort of lost. So one way we can get around this is, hmm, here we want this class, specifically we want to collect all the errors. Is there even a, that's a good question. Is there a collect errors to just drop the okays? Good question. Does not look like there is. There's so many methods on iterator now. I specifically want, no, doesn't look like it. Okay, so we're gonna have this just return an okay value. We don't actually, we don't want to collect the okays because the only thing that's that comes out of this are all the machine sessions. And we're already setting those directly in by getting a mutable reference to the machine. So we want to collect all, oh, specifically, I want to be able to use the question mark operator in here. That's the reason this has to be an okay. But then we can map, we can map all the values, I guess, filter map specifically. And I want r dot, I always forget, what's the thing on result to give you the error back? Is it just an error? So, sorry, so filter map, in filter map, we're now given a result. So the outcome of running this closure on every machine, which is going to be currently a result where either you get a failure, like you get an error, or you get an okay blank. And we want to only keep the errors in case there are any. So I want to filter, I want to filter out all the things that are errors. You recently saw res iter. I don't know what res iter is. That seems nice. I don't think I, okay, that's a good catch. Although in this case, I don't know if I want to add a dependency on that because it's so yeah, it's error. It's like, I can just do this, right? And that that's I can do this line instead of importing that crate. If I did this in a lot of different places, I totally agree that we could use this externally. That's right. So this is one of the places in which our API is going to change because now notices that this f this user defined f the setup routine for every machine is now running parallel. And so this means that it can't just worse the yeah, so the machine set of box remember was just normally an fn was just a regular function. Now we actually need to require that that function is sync. We need to be able to call it from multiple threads at the same time because that's effectively what rayon ends up doing. So we're going to add a bound that this is a sink. And this is also going to sink. Now let's see what that does. So now it cannot in where 295. Oh, interesting. I guess we will have to say that this returns our results. That's the result we use here again. That good enough for you. Cannot infer type for C. Oh, right. Yeah, so this is it. It needs to know what we're collecting into in this particular case. So the problem here is what do we want to return to the user? If you get multiple errors, like now it's going to try to set up all the machines in parallel, right? And if it if it does that and say two machines fail, what is the correct thing for us to do? Is the correct thing to just abort? Probably we probably want to run termination as well. But what do we do return to the user? Because we only have one return value. But I wonder whether failure has something like this. Does it have something to let us combine multiple errors? Not easily, I think. Like for specifically for error, can I can I create a causes? It doesn't look like it. That's too bad. I specifically want a way to take multiple errors and turn them into a single one in some useful way. But it doesn't seem like that's easily easily doable. So I think what we're going to do is we're just going to keep track of all of the errors that we come across during setup. I don't think the maybe that's important. No, so that we don't actually need to keep like a map of which group they're in and such because that's a part of the context of each error. I think what we're going to do is map this this error with a context. No, we already have the context. Yeah, so never mind. Right, so I think we're just going to collect all the errors that we get trying to set up all the machines. If there are any errors, then obviously we're not going to call the main run function and then we'll fall through and go to termination. And then this okay at the bottom, we're going to call only only give okay if there were no errors. Okay, so I think that should work fine. So in this case, this is just going to be errors.extend. This means we also don't need the class. Okay, let's see what this does. So now gives an option failure. That's not true. Let's see is not an iterator. Am I missing something? This is definitely an iterator. Someone's seeing something. I'm not like, why is this? Yeah, so it maps it using filter map. And it's saying this is not an iterator. That's so strange. Let's see. See, these type get really long. It's really sad. Yeah, so it filters it with something that returns an option error, which is corrected. That should mean it's an iterator that outputs things of the type failure error. Why is it saying it's not an iterator? Interators not implemented for filter map required because the implementation of into iter. Let's look at what the for rayon, if you have a filter map, what does that require? So iter, filter map, it implements parallel iterator. That's why. Right, right. I don't know how they're, what the intention is. So I mean, we could of course just collect and then iterate, pair extend. That makes a lot of sense. Yeah. Yeah, that makes a lot of sense. Yeah, so the observation here is that this iterator is not a regular iterator because it can iterate. It's a parallel iterator, right? It could generate things in whatever order and generate multiple things concurrently. So saying extend here does not actually, it's not actually helpful, but pair extend maybe. It's like magic. Thanks. That's awesome. All right. So now we're connecting all the errors and then I guess we want, after all the setup, if there are no errors, then of course we're, there's no reason for us to run the main procedure. And then we are going to terminate just like normally. And then down here, we're going to do if errors.isnt, then just do that otherwise. And so here's the question. Now, I guess if errors.len is one, then this is trivial. And we just do errors.let's just not do that special case. Yeah. So the question is, if there were multiple errors during setup, what do we return to the user? I don't actually know what failure says that you should do here. One way we could do this is to just change them all, like make a long causal back chain of the errors that have happened, but that doesn't seem quite right either. I wonder if error is implemented for, or fail is implemented for like, VEC of error. I don't think it is. Anything that implements display, which VEC does not, anything that implements standard error. See, I liked the dark theme for a while, but it gets in the way. It's also not particularly helpful. Any suggestions on how we should expose that error? I sort of, what I really want is, like, I want to return something that's just error of a VEC of failures. Like, I could totally see this being useful. And then, or I guess dot into, like just for failure to be the derived fail for, derived fail for struct errors, VEC error. Yeah, but that seems so weird too. Like, I don't really want to introduce a new type here. But yeah, maybe that is the thing to do. I have this from a long time ago. I sort of want an error collection, right? I'm a little surprised that this doesn't exist. It's just unhelpful. Yeah. So the other thing is I could just do the debug of all the errors and turn that into one. But also just, like, feels wrong. I would also lose all the context. That is a very good question. All right, let's just do, I guess, error collection. Multi error? Sort of feels really weird. Right? Like, this doesn't, this doesn't seem right that I should have this type in my library. 17. What do I need to macro use? And wait, what? I think this can derive display. That's not very helpful at all. Let's see. So what does failure say for using your own types? So where's the this one? Failure derive. Yeah, that's a little bit unhelpful. So failure derived gives us fail, which is just the proc macro. So I'm just not allowed to do this. It seems like I'm not allowed to derive fail for this type, because it requires debug and display. And why can't I do display here? Because if I derive macro display, well, I guess I'll input display. Yeah, this seems this seems very unnecessary. See that link. Oh, I see. So you, right, can I use this without having an enum? Like, can I use this here? Because that allowed? Or is it going to complain? Or like here, I guess? Oh, that's also nightly only. It's all right. I mean, it's fine. I can just implement display for for this. It just makes me a little bit sad. See, now I need to do all this stuff as well. Oh, and that's the wrong kind of error. I think we should probably ping boats about this. This doesn't seem very ergonomic. So display for this is just going to be self dot zero, format. I don't think Vex this implement display this way. Hmm. Okay, you know what, we're going to do this instead. I'm going to not do this. Because it makes me sad. Instead, I'm going to say otherwise, into it, or actually, we're going to do better. We're going to do this dot next dot map. So this is going to mask some of the errors. You need. Yeah, but the problem is the derive is also nightly only. And I'd rather not do that right now. Yeah, sorry. So if you didn't catch what's going on here, I'm basically saying, just take the first error if there is one, which like, I guess we should. This will only expose first set up there. And then I'll probably ping boats about this because I feel like there should be some standardized ways to expose collections of errors. The derive totally works on stable. I was giving me some sadness, but maybe I'm wrong. Yeah, I'm going to leave it like this for now, just because it's not seem worth having my own error type exposed that way. It just doesn't seem necessary in this particular case. Okay, so that compiles works. Great. So now in theory, it'll run all of the setup things in parallel. Let's see if that actually works. Go back here. We're gonna in the security groups, we're gonna go delete the old ones. In fact, I'm going to delete all of these except default, maybe delete those just to see that the library now cleans up nicely after and for the key pairs, I will delete the old one it made. What font am I using? That's a good question. I think right now I'm using noto. Yeah, noto mono. It's actually really nice. It also has decent support for like text based emoji, which is surprisingly useful. There are a lot of command line tools that now will like print status updates and stuff using emoticons. So that's actually been really nice. So specifically, you also need to install this noto fonts emoji, and that gives you mono spaced emoji as well. So I found that to work really well. All right, it should be running. Let's see. Does it create a key pair? I guess, does it create a security group? Yes. Does it create a key pair? Yes. Does it create a spot request? Not yet. How about now? I feel like that's a lie. I feel like it did. Instances? Instances? Oh, it crashed. Dependency violation. Oh, that's a little bit annoying. Okay, so the it's complaining because the security group has a rule associated with it, and you need to delete all of the rules before you can delete the security group, which is annoying. But okay, I guess we'll do that. So let's close some of these tabs. Oh, that's one too many. So specifically, we're going to need, where is it? So it was, they call it like grant allow authorize, authorize security group egress, so security group ingress, revoke security group ingress. Is there a way to remove all of them? Because that's really what I want. It doesn't look like it. I guess delete security group. Delete security group. If you attempt to delete the security group that is associated with an instance, huh? Okay, so this might actually be different. This might just be that I'm allowed to delete the security group, but not until all the instances have actually been terminated. So this suggests that we actually need to wait for termination, which is a little bit sad. But I guess we could do that. I don't really want to do that. It's a little bit sad. So I think we will comment this out for now and have it not do clean up. And we'll deal with that later. I specifically wanted to see if it did set up in parallel, which I guess it must have, because it did get to tear down. And it did print all the things, right? No, it only printed one. Okay, so this is a little damning, did not print the date of the other ones. So remember how we have three clients on one server, so we're expecting this command to be run three times, right? One for each client. The fact that it didn't print those is a little bit disturbing. But they might just have been because it like crashed before it printed them. It seems a little bit weird, but so this should, there we go, for instances with some temporary key, let's see. Arguably, I should have done logging first instead of after because it would have made this process a little bit easier. In fact, I'm going to do that quickly. So log is going to come next running anytime now, AWS security. I wonder whether we could just disassociate them instead. Can I like remove the security group without? No, it doesn't look like it. I was thinking whether I could like remove it from the instance without terminating the instance, but why is this still running? What is it doing? Find our trusted GDP again. Where is it stuck? Right, so this is what we saw earlier as well. It's like trying to connect to a server and the server is the machine is like not yet ready. I wonder how to get around this. It's basically a race where it tries to connect. It seemingly, I thought this was a DNS race issue, but it looks like it might not be. That's two dates. That's interesting. So we got two dates. Notice that this time, this is also sort of interesting. We got two dates before we got the server, but then it failed on the third one. Failed to connect to SSH port. Okay. I think we need a better timeout mechanism here. So Rost TTP stream. Let's see if there's a connect timeout. I think that's what we want. So we're going to do connect timeout. We are going to do, we're going to have a shorter timeout and let it retry more times. This takes a duration. So we're going to use time. And specifically, this is the observation that sometimes it like tries to connect and just takes forever to try even though the machine is now started. So if it instead stopped and tried again, that would work better. So we're going to do a duration of from seconds, I don't know, like, should try for, should try for five seconds, 10 times, right? So that brings us to like about a minute. It will try to connect, right? Let's try that. I expected, wait, that's terrible. Connect is generic and connect timeout is not why you do this to me Rost. Oh, that's so sad. That's so sad. Well, I guess we will have to do this then. Yeah. So okay.  Whereas connect timeout takes a socket address. So this means that we sort of need to put two socket adder, if you give it like a DNS name, for example, it's going to iterate over, or if you give it an interface name, it might iterate over multiple interfaces, or multiple ways to connect to that particular address. In our case, we actually might not need this. We might be able to just take a socket adder because we do have the IP address. Let's do that. Let's have it just not be generic and instead just going to take a socket adder from Oh, it's going to complain in the place that we connect. So we're going to go to lib here. So currently, notice that we're doing this like string formatting, which is actually a little bit silly, because we have a public we have an IP address. So why don't we just make one. So let's see, can I so we're going to do net socket adder. In fact, socket outer v four arguably, but no, let's do this. Okay, so we're going to have a socket adder new. We're going to give something somehow we need to get the IP, and then a new and that address this IP address needs to be parsed out of the public IP. Which I think we can just do with parse. I think this supports parse, right? Yeah. So we can do machine dot public IP dot. And in fact, we can do this. So if the public IP address is not an IP address, then maybe AWS lied to us. So machine IP is not an IP address. That seems really weird, but okay. Wait, what? The parse is so from string gives you an adder parse error. Why can't it resolve this? That's interesting. Does this like not implement standard error or something? No, it does. Alternatively, you can call local adder on the TCP stream you have. I don't have a TCP stream. So this is at this point, we're trying to connect to a new machine, right? So this adder is not a bound adder. This is just we want the address of the local machine we're connecting to. And specifically, I have the public IP address and the port. So in this case, I want to parse the public IP. And it's telling me that telling me that it can't resolve the parse of that. Do this without context, just to see. So without context, it's fine. That's a little bit disturbing. So I guess this is as a result of the result x from failure, right? That we saw earlier. So this requires, so for whatever reason, the parse, that's going to give me all sorts of things, right? Yeah. Oh, I see. Okay, so I think, I mean, I could help the compiler by doing this and then giving it the parse socket adder error, whatever that was called. Where do we find this? So the from string gives us one of these. Right, so I can help the compiler and tell it that the error of this is expected at most one type parameter parse. Oh, I guess this just needs to be socket adder. So I could reverse this one actually come to think of it. I could say, no, wait, I'm not parsing into socket adder. What am I talking about? I'm doing this. So I might as well just do this from string of that. But that's the same as this. Let's see if it gets happy about that. Yeah. Okay, so whatever, for whatever reason, it can't do type inference through context here. So a little bit sad. But okay. So here what we're doing is we're parsing the public IP address into well, an IP address. We're then connecting, we're creating a socket adder for the place we're connecting to, which is going to be the IP address now in a structure format and the port that we're connected to, which is port 22. And then when we connect, we use that socket address to connect with the timeout now to the server. I feel like I want to make this shorter. Okay, that's exactly one minute. So let's try that. So in theory, this should now do all of the things that we want. Except it still doesn't do cleanup, because cleanup is annoying on EC2, apparently. Really surprised that they don't let you do this. Or that there isn't a nice way to do this. I guess we could just wait for the interfaces to terminate. It's a little bit sad. I guess we would have to, is there a wait? I would have to describe all the instances. I basically want to describe all the instances until this head is empty, but I'm going to not deal with that right now. All right, so we should have some instances again. In theory, let's see. Great. Now is it going to connect to all of them? That's the real question. In fact, let's do a while we're on this. Someone got mad at me for not did committing lots earlier. So I'm doing it now. We're going to comment this code out. Don't, by trying to clean up before instance termination, which we're going to have to fix later, fail to connect again. Feel like that. Oh, connection refused should not actually make it give up. I'm so confused. Why is this not trying for longer? I don't think that was a minute at line 21. That's a good question. Oh, see, it's not actually a minute, right? So the connect timeout will connect and then it might get a connection refused really quickly. So I think really what we want to do here is we don't want to try a certain number of times. We want to try for a certain amount of time. So we want to do something like here, let's get instant in here. We want start to be instant now. And then we want if start dot elapsed is more than duration from sex. Yeah, so if it's more than a minute, then we do not or then we try again, essentially, quite off topic question. I don't have any recommendations for where you would hire coders. Let's see. Let's see if this runs. So basically, the change we made was instead of retrying a certain number of times, which could go really quickly, right? If you imagine that you try to connect and you get connection refused each time, arguably, we should put a weight here. But I'm going to not do that for now. And so instead of just trying a certain number of times, which could go really quickly, we're going to try for a certain amount of time, connection timed out. This is definitely not right. Oh, yeah, let's not silly. And then we want this, of course, to sleep for a little while, so that we don't try really quickly duration from sex. So we're just feeling really nice to the API and not spamming it. Someone suggested in the previous stream that I did that we already have some loops that are like, while it's an error, try again and we should probably have those sleep. But for now, it's fine. Some of you called the error that I made back there. But here, I have this the other way. So if start to the laps was more than a certain number of seconds, but of course, if it's less than a certain amount of seconds, we want to try again. If it's more rushing an error, see if this doesn't work a little bit better. Arguably, we could now make the connect time out less too. So we should now see three dates and a hostname. And then all the IP addresses, if everything works correctly. The instances, good first question. Yes. See, this is why I don't want to wait for the instances because it takes a while for them to shut down. So I can annoy anything to wait for because you know that they're going to eventually shut down. Just want to like remove the security groups when they finish shutting down. In fact, I wonder whether terminate has a way to do this. Fantastic. All right, I did the right thing. Good job team. Okay. So now we can first just commit to the changes that we made to the socket connection just to keep our changes nice and cornered off. This is going to be a little bit of a pain. But essentially, this is going to become it is a little bit fun to to manually edit the files because it's like a gamble. You never know whether you're doing it even remotely correct. Like all it takes is the wrong space. But I think, I think this might be right. So this stayed the way it was, this stayed the way it was. What's a concrete example of to task tsunami would be used for. Okay, so the use case I have in mind is I'm building a research prototype of a new database and I want to do benchmarks on it on large deployments. So have a server or cluster of servers and then have say 20 client machines all hitting it with load.  And then I want all the climates to start hammering the servers for a certain amount of time and measure the latency and throughput that they get. And then I want to tear it all down. And I want to run this on spot instances and not longer running instances because most of the time I'm not running an experiment. And so I only really want to pay for the time when I'm actually running one because that means that I can run larger experiments. Other examples of this would be if you run a compute heavy job that you just want to distribute over if you want to distribute like some large compute tasks similar to what rayon does over threads, but instead have it distribute over machines. So one thing you can imagine is like a rayon on EC2 that use this library, right? Like a not a parallel iterator, but like a multi machine iterator, assuming that your types are serialized and deserialized or something, right? They could build on top of this, spread your code across many machines, run it across many machines by spinning up EC2 instances and then tearing them all down and giving you the result. That would be really cool. That is a decent step like of complexity from what we have, but this is sort of a building block for that.  Okay, so this we want to commit as more robust SSH and now all see. So adding rayon to our code to have parallel setup was this much. It's not that bad. Not that bad. So there's still the question of whether we want to set up machines in parallel. So the way we would do that is if you look at rayon, I have way too many tabs now, just get rid of some of those. If you look at rayon, there's sort of a lower level API you can use with scope. So inside of scope, we could use we could create a new scope, which is essentially sort of everything managed by one thread pool.  And in theory, those tasks could be things like doing parallel iteration. That's a bit of a low level interface. I don't know if you could do this just with the iterators themselves. That's a good question. Like, can I have what happens if I construct a parallel iterator inside another parallel iterator? So think of this as actually let me commit this first. Do machine setup in parallel within each group. So if you look at the code down here. So currently, our pattern iterator is just over all the machines in each group. But that means that each group reach set is going to be done sequentially. So ideally, we sort of want a parallel iterator over this. And then within that parallel iterator spawn more parallel tasks that are going to do this setup. That's going to be a little bit tricky to get set up. So I'm going to not do that for now, in the interest of like covering more integrations with other libraries. Yeah, so I think what we're going to do actually let's check our to do file. So Rayon has been done. This has been done. This is mostly done. Okay, so I think we're going to do scope guard would be really nice. But I think we're going to do logging next. All right, logging. Rayon can go away. All right. Okay, so there are lots of logging libraries for Rust. I've had particularly, I've been particularly happy with this one called slog. It's fairly low overhead, you can, especially most of the logging calls are removed entirely at compile time. So for any logging level that you're not using, it gets removed entirely from the code. You can of course have them compiled in and change it at dynamically at runtime, but you can only change down to the level that you were compiled with. So for example, you can have all your trace events compiled out of your release build. I believe Rayon allows up to a number of CPU cores available for running tasks. Since we're spending time waiting for AWS instead of doing CPU intensive tasks, should the default number of threads available be overridden with more, would make standing up more instances faster? Yeah. So the observation here is similar to one we made earlier of Rayon is sort of intended for compute heavy tasks, whereas this one, we're really just sort of cheating by using Rayon. We're using it to run multiple blocking tasks at the same time. So we could hear use a thread pool or something instead. The something came up earlier where someone mentioned that shouldn't we just use async IO for this, like Tokyo and such. The concern there is basically that the SSH crate we're using, so this SSH two does not support async IO. If it did, then we could totally do that. And that's probably the better way to do it precisely because none of these are compute bound. As far as I'm aware, there isn't a good SSH library crate that is async. Basically, you have SSH and SSH two. Why is SSH two? Yeah. So this one just binds lib SSH. So this just binds lib SSH, but the channels here are just channel reader, for example, they're just read, they're not async read. Yeah. So the proposal, you're sort of proposing a workaround here of why don't we set the number of threads to just be higher for the pool, given that we know that it's not compute intensive. We could totally do that. It's a little bit unclear what we set that value to, because we also don't really want to send it to the number of machines we're connecting to necessarily, because imagine that you're spinning up 100 machine cluster, you don't want to try to do 100 SSH connection setups in parallel, which that would do. So again, I think using Tokyo for this is really the right solution. So yeah, so I've seen Thrush, but the problem with Thrush is that it's still actually maybe now it basically doesn't integrate with SSH agent and such. But now that we it's a good point. Now that we have this like private key file ourselves, we might get around doing this. So on a session, can I get a channel? I can. Yeah. So it might be that we could use this now, actually. Yeah. So I've seen demos of Thrush before. It is really cool. Basically, the reason I didn't use this in the first place was because the way you authenticate here is a little bit tricky. The API is also a little bit awkward to use, but yeah, no, that's a good point. Let's sort of sidebar that, but that's a good point. I'll leave that for slightly later. Now that we sort of have parallel setup working, but I agree, it could totally be improved this way. There's also a question of this whole change now to Tokyo and futures. So the both Tokyo 0.1 and the fact that it might change again for 0.2, future 0.2, async await that sort of may be coming in likely and not too long, given the progress the boats has been making. I think it might be a little bit premature for us to build this async right now, but maybe like, in fact, maybe a good point would be in a subsequent stream to convert the entire library to be futures based. I think that could be a useful sort of standalone contribution to just in terms of people who only want to learn how to make things async. We could totally do that. Okay, let's do slog for now, but that's a good point. Let's go here to do async instead of rayon. So right, so slog is really useful. It also lets output lots of, like you have a lot of control of the output, you can like log to either things that are used by end log, for example, the ones supported by the center library, you can output to syslog or journal D, JSON, whatever you want. So it's just a very versatile login grade. Yeah, so this is the example of it removes trace and debug in release builds by default, you can change this, you can dynamically change the level, but not beyond what it was just compiled in. At a very high level, what you do is you have in slog is you have drains. So drains are the things that you write commands to, sorry, that you write logging things to. And so in our case, we will probably just want really straightforward logging to the terminal. But probably what we'll do is, where's our tsunami builder, is that our builder will let the user specify a logger. And then maybe we provide a default one. So in our case, we're going to have, so we're going to have, sorry, sorry, correct the tumble. Slog version, what, 2.1.1. And I guess probably slog term 2. Yeah, so slog is split into multiple crates. There's one, there's one for slog itself, which contains sort of most of the APIs. And then there are crates for the different kinds of outputs and the different kinds of things to generate. In our case, we just want to like print a terminal. That's all we want. So we're going to take slog term version 2.3.0. And then at the top here, we're going to get into these, arguably, we don't actually need slog term and theory and are part of the library because everything in everything in slog is sort of exposed through this logger API. So they have, where's the traits logging drain?  And debug takes it L, which is a logger. So if you look at log, log. Yeah, so log takes an slog logger. So it's one of these. And logger is generic over the underlying drain type. In our particular case, what we could do is just have our library take a logger of some type. So we could hear. Yeah, so there's sync and send and they implement drain and everything. So basically, in our builder, we would just take in whatever logger the user wants to give us, and then we just use that logger. And this means that we don't actually need to include term. All we need is slog. In this case, because we want to provide a sensible default, we will also have log term. So we're gonna have a log. It's, it could be an option, but then we need to like unwrap in lots of places. The way you get around this is slog also has a drain that is called null. So it's called discard.  Instead, this is going to be an slog logger. It's a little bit sad that this is. I feel like this didn't used to be generic. It's a little bit sad that it is actually. Yeah, so that's the one I want. I think I just want the erased logger for this thing. So I will do this. So basically we're erasing the, we're erasing the drain type. This is going to decrease performance a little bit, but it means that we don't have to make our builder generic, which would be really annoying. So this is going to be the standard sync arc. And we need, this is going to be what even is this? Is that in the standard library? No, there's no way, right? This is something they have somewhere. Yeah, okay, great. So slog that and what's their never type? Never. I feel like never is also their type. Maybe I'm wrong. Is there a rust never type? That's not, I mean that is never, but that's not the one they're using. Why is never not linked? It's a little bit sad. So logger, where is this never type? Type definitions, enums, am I missing this somewhere? There should not be that. Well, I guess slog, let's try slog never. Let's see what it complains. Missing log. Yep, apparently it's just in slog, it's just hidden or something. Yeah, so our default logger of course is just going to be slog, log, slog discard. Right, so by default, it doesn't log anything. And then we could say on the tsunami builder, you can do a, with, this is one of those places where we might want to take a look at the API guidelines for the builder pattern. The thing is currently, there's some discussion about, about how builders should be constructed. Essentially, should the builder take a mute self? Or should it take a self? Should it return a mute self? Or should it return a self? Right. Like both of these are valid builder patterns. They both have some advantages and drawbacks. I think the guide currently recommends that you do this. But it's essentially so that you can chain builders or chain builder setup. I'm going to not do that right now, just because it doesn't particularly matter in our, in our case. Because if you look at the example code, the functions you call are fairly verbose. You could imagine that we instead did like this. But I don't think that actually adds much in our case. Here, you can also see the reason why some people argue that you should take self instead. If you take self, then we could do this, and this, and this, and have run consume self. If you return mute self, then now you can no longer have the run method or your build method consume self because you still have an outstanding reference to be. So I think we're just going to leave this in the simplest form for now. This is worthwhile to know that this exists. This D, I think it's fine for us to make generic over the drain. It's going to be an S long drain. Gee, that's a good question. I think actually what we want is a logger. Let's do a logger instead. Ooh, that's why I don't need that's the default. I can just do this. Great. Never is defined as a hidden internal type within S log. Yeah, it also says that it would eventually be replaced with exclamation mark. Yeah, so exclamation mark is the sort of Rust never type, but it doesn't really have its own type yet. It's a little bit iffy. But yeah, that makes a lot of sense. It also turns out we don't need it. So notice I read this wrong. This is an equal sign. So this is a default value for D, which means that if we just create a logger, it will use this type by default, which is fine for us, which means that I don't need to refer to never. And in fact, this is probably why never is hidden because you shouldn't generally have to refer to it. It's only if you use this type and this type is used by default. So we're just going to take a logger. And I think what we really want here is a set logger log is going to be a log logger. And again, we're going to require essentially that the D is the default value here, right? And so here we're going to do self dot log log log log. Actually, you know what? Now we'll provide a default. That's fine. So here you can imagine that we require the caller of us to be the one to like create a terminal logger and then give it to us, because I think this is so common. We're just going to provide that code for them, which is basically this code. That's a good question. Let's find out. Nope. That offload processing to another thread, because they hinder performance. I don't particularly care about that. I think. I think realistically, what I want is I'm fine with it just like taking a lock or something. So that okay, so the reason this exists is because imagine you have multiple threads that both writer logged to the same logger at the same time, you need some way for them to exclude one another. There are many ways to do this. Like you could put a logger behind a mutex. So if we look at drain here, you'll see that drain is implemented for this where it sends sync, save, unwind, save. Where is the views? Oh, it's fine. Okay, we're going to create a terminal logger. And in our case, it is fine whatever it is. But let's just do this. So basically, the saves users of our library from having to depend on s log or s log term, and it saves them from having to find this code and write it and like write it in their code. Oh, here is you can set a bunch of tags that are going to be included on every line printed by this particular logger. So this is really useful if you're about to enter a scope, for example, and you want every logging statement inside of that scope to all of them contain some given tag, then you could create a logger that is sort of contains another logger and that outer logger is going to have an O of say like, I guess like, you can imagine if you had an iterative computation, you could say iteration is like one. And now everything that uses this logger would actually, I guess, so let's do, say we have this is going to be easier to explain this way. So now you have a logger, and then you're about to do some loop. So you're going to do like four I and zero to 400. You could say you can make a new logger in here that's this dot new with an O of iteration to I. And now in here, if you now use like debug, log, and then something, then now this is going to include iteration is equal to I. Right. Whereas once you go back here, this will not include iteration is equal to I. Right. So you're creating a new logger that is the old logger plus some additional tags. So this is really useful for new scopes. In our case, we will not set any such options. We're just going to set self log is equal to and our example is going to be the doctor. I guess use. Yeah, it's really, it's really convenient. I found this guard. Oh, yeah. Logger, generally creating loggers is a pain. You need to go through this like song and dance of a drain is not a logger, a logger writes to a drain. And this is because you can have different formats, right? So the drains are just the output places. And then you have a you have a logger that uses a drain, or you could have a logger that uses a logger that uses a drain. This is the example for when you add tags. Oh, it's because I removed the async. I think. Okay, so the problem here is that this drain is not currently thread safe. Whereas the default remember there's a default D here, that default D requires the drain to be sink, which an S log term is not because if you try to write to a terminal for multiple threads at the same time, you get interleaved results from memory, this may no longer be true. You can just do this. Like it, it knows that a mutex is safe this way. I may be wrong. Expected mutex drain error. Okay, so what we're doing here is we're creating a terminal logger. We're hiding it behind a mutex. And then we're using that mutex as the target of the logger. So now every time it tries to log something, it will take a mutex. But we're okay with that. Like we don't really mind if it takes a mutex because this is not performance sensitive code. If you did have performance sensitive code, you might not want to use a mutex here. You might instead, oh, you might instead want to use like S log async, or there are a couple of other S log based things. So S log async essentially it spawns a separate thread for you. And that thread is going to do the logging. And then the anything that logs is just going to send on a channel a message to that thing to print to your terminal. Whereas for us, a mutex is fine. All right, so now this compile this code compiles. And now we start adding our debugging. So in S log, I don't want this. So in S log, the way you do debugging is essentially that you use these macros. You can ignore the ones down here, they're mostly for namespacing reasons. But the ones you want are info, debug, where's warning, warn, trace, crit, an error. So the ordering of these goes from the most verbose to least verbose, and this is fairly traditional, is trace, debug, info, warn, error, crit. So the idea is that if something is trace, it is probably not useful information. So this is where like, you would use a trace on every iteration of a loop, for example. And so if you were actually debugging something really low level, you would use trace. And just because it gives you all the information, but normally, you would not want all that junk printed to your screen. And perhaps more importantly, you don't want the performance over it. So this is why trace is usually compiled away in release builds. Whereas crit on the sort of other extreme is your program will probably not be able to continue. And so in general, the visual style it presents these in is also representative of how important they are, like a crit will be showed more boldly than a trace would be. In our code, we probably mostly use info. So info is the thing that is printed by default in every build. Debug is the one that's only shown in debug build. So in test, for example, and not in release built by default. So let's see, we're going to do something like info.  So here, like, hello world, and then return. So notice that we're not actually not doing anything. I just want to show you what this looks like. So now we just added a single info statement. There's nothing else there. And notice how we got clear this show you again. We just got a single line that says info, hello world. Great. So logging works. So what we want is something like connecting to EC2. Now, arguably, this is not important information because it will always do this. So let's do debug. In release mode, there's no reason to really print that message. And then I guess here, self.log connected to EC2. Another thing we'll probably want in our builder, by the way, is for ability for the user to choose how to authenticate to EC2. But we can deal with that later. Down here, here, for example, is an example of something that might be really useful if you're debugging is the name of the random security group that our library chose. And so this is something that's like a trace level of it, right? Normally, you don't care about this information. But if you are debugging, you might care. Now here, we could either do, so it supports sort of print up for format style things. So we could do this, and then give it a group name. However, instead, what we can do is we could put a semicolon, and then you can give tags. So you could say, creating security group. And then it is a name. It's going to be group name. So the way this is going to be printed is it's going to print, you have a hard coded region. Oh, sorry. Yeah, you're right. And region. Good catch. Yeah, so you can add additional tags with sort of a key, a bunch of key value pairs that will be printed after the line here in the terminal output. This is not terribly important. They're not distinguished very well. But in other kind of loggers, you should upload this to YouTube. The part one is actually already on YouTube. And my plan is to upload part two to YouTube as well. But thanks. Yeah, so the tags are things that you can in the terminal, they're not terribly important, they're just like printed after the main, the main statements. But in other kind of logging outputs, like if you log to like the system D journal or something or syslog, then those tags are actually reported as tags. So they provide slightly more helpful semantic meaning. So in this case, we're going to the message is creating security group and the name of the group is printed. And this is a trace event because normally you don't really care about the name of the group or the in fact that one was created. So we're going to do that. And similarly here, once we get the group ID, the fact that it was created is not terribly important. So it's going to be a trace level event. And then signed ID is going to be and similarly down here, adding the rules and important, but here we can add another trace like adding SSH access to security group.  And you never know when they might be useful, right? It's super useful to just be able to change a compile flag rerun your thing. And now you get lots more information. And that way, you probably don't have to go back and add lots of print statements because you will have sort of the basic flow of the program already printed for you. And then here, our last debug was connected to EC2. I think one of these, this could also be a trace. So think of debug as the things you normally see during development and of info as the things you normally see during a release. And trace as the things you see when something goes wrong. So normally, in sort of normal operation, we don't even during development, we don't care about the fact that it creates a security group, that it creates the key pair, we don't really care about this thing, creating key pair. Here, I think created similarly here, we can include the fingerprint of the thing that was created. Fingerprint is going to be what was it rest up to fingerprint, I think. Here we could also do wrote key pair to file. And then here's another example, the tags they use in sLog by default need to implement display. That's how or actually, it needs to implement sLog value, I think is the technical this thing, any value that can be printed, and it's implemented for a bunch of the basic types strings, etc. And for display. But a lot of times what you really want to do is you want to include a debug thing. So you could imagine that you like do something like this, which I did for the longest time. And you do not need to do this anymore, you can now do this. If you prefix the value of a tag with question mark, it prints the debug version of that value. So in this case, actually path may implement display. So this might be fine. Yeah, so where the key pair is written is also not normally important information. Here, though, we're about to start spinning up the machines. So here debug issuing spot requests. Here, we do want the trace to be down here somewhere. Let's see here. So a debug level event would be like, we're spinning up all these machines, whereas a trace level event would be we're spinning up this specific type of machine, right? issuing spot request for this. And then we're going to take a tag, which is number of machines. It's fine. And in fact, in here, we could even do activated spot requests. In fact, this is something we might want to do in parallel to issue spot requesting. No, it does seem to be a little laggy. Yeah, it's unclear why the stream is lagging. Let's see. It looks like it's no thing. This is looks to be twitch. Sorry, I'm just trying to see if I can fix the lagging in the stream. The audio is good, but the video is lagging. Yeah, this is usually the case if my CPU is not doing lots of things. So no, it's unclear. That's unclear. Wait, let me check if it's my internet. Why is the internet being unhelpful? It's a little bit unclear. Is it better now? It seems to have recovered. Yeah, it looks like it's about back to being good now. Can anyone confirm? At least Twitch is now saying that the stream quality is excellent. So maybe there's hope. Okay, let's continue that. All right, so here's another example of us wanting to trace. We might want to trace every single spot request that we get back as well. So in this case, activated spot request here, we that's a good question. So currently it's fine for this because this is currently all sequential and not in parallel, like we sort of may want it to be eventually. It's fine for us to hear just print the spot request ID. But you could imagine if we run this in parallel, you'll get like interleaved logging results or logging lines where like, you'll get this for one group. And then you'll get one of these from another group because they're just all happening concurrently. So we're going to have to be a little bit careful here. Add more info. But we can avoid including that information here because we know that it'll just be the spot request will be for these, this issuance. But that will not be the case in parallel. All right, so here, our last debug was issuing spot requests. Our next debug is now going to be debug self log waiting for instances to spawn. See, here we could put a trace is again not important. So this is the case where we talked to the API and the spot request has not yet been issued, even though we created it. And then we just retry. But if if you actually you're debunking a problem, this seems like relevant information. So that's generally what you should think of traces being for. Similarly, here, this is something we'll want to trace. So this is if if any of the spot requests do not have, if any of the spot requests have not yet spawned their instances, and we're sort of rechecking again, then that's something we want to trace. Some fact, let's do this here. So specifically, if machine is not open, I guess we could checking spot request status. And then we will log in trace mode, we'll just log the state of every spot request instance at every step in the spot request of state. And then when it is ready, down here, down here, if it's active spot request satisfied with spot request satisfied, and that's going to be the spot request ID, instance requests, and the instance ID, which is going to be zero dot instance, notice that these are options. So instead of unwrapping them here, and then we could have this be another to do the security group can be set to allow all traffic from instances inside the security group. That way, the side of box difference instances can still talk to each other. Yeah, that's true. So currently, make security group policy allow not be sider dependent. This is sort of related to setting up a VPC. So if we set up a virtual private network for all the machines we spawn up, then the security policy would also change because then we would just use the sider of the VPC that we spawn. All right, does this currently compile? No, it's being mad at me. 155. Path is, oh, thought it was implemented for display. Am I wrong? Oh, is it like a bit? I think that may be like a prefix you need to use. Path cannot be formatted with a default formatter. Display is not implemented for path. Well, then I guess we're using debug. I'm really sad the Rust format doesn't format long macros, but I know there's some work going on in trying to fix it. Why does this work? It thinks I'm like moving all these values, which also seems a little bit unfortunate. It seems like a bug in slog. I don't understand why that, why it's complaining about this. So use a value name. This is going to have to be this. So basically, for some reason, this macro is built in such a way that it actually moves its arguments into some closure. So I need to give reference to it. Otherwise, I wouldn't be able to use, for example, name down here. This is Rust. Yes, this is Rust. It's not Rust the game. It's Rust the programming language. Trust me, this is frustrating for both players of the game and users of the programming language. It means it's much harder to find things in search results. 247 sir is moved because we move it where? Oh, it's because of the trace up here somewhere. That's my guess. Do we have a trace here somewhere that like ends up moving? Fine. And this looks fine. Oh, I bet you I know what it is. It's the move out of self. Yeah, I should have just actually read the error message. Yeah, it's terrible to search both for Rust, Dart is the same. Go is the same. Like all of them you need to add Lang to now.  So I recently switched to trying to use DuckDuckGo instead of Google. And it is terrible about the Lang suffix. It does not realize that it should also search for things that are related to the programming language, but without the Lang suffix. It's not great. It's not great. Yeah, okay. So the problem here is that self.log is like being moved into all these closures. So we're just going to do this, which means that now we can do this. Now we can do Wait, so that means maybe it doesn't think we're moving these this. Maybe I'm overly optimistic removing all the checks, but yep, I was indeed.  There we go. All right, getting there slowly but surely 109. This also is not supposed to take the group name. This is not supposed to take group ID. This is not actually you can take that. That's fine. This should also use log. Oh, I see this here. We have the same problem. In fact, given that this all consumes log, how about we just do let log is equal to self.log. And then we can just get rid of all of these and just have them use log. Maybe it should even be that was easier. All right, so now how far did we get? Okay, so down here, spot request satisfied, spot request not ready. And here we sort of wanted to say spot request satisfied, right? So that is there. Then we stop all the spot request is also not something that a user normally cares about. So we'll make this a trace. Terminating spot requests. It's fine. And now we get to another interesting part, which is here. This is when we are waiting for all the instances to have spawned. So I think right here, we're just going to do also not something the user probably cares about. This is Rust, the Rust programming language. This one. It's great. I highly recommend it. Yeah, so the it's a little bit unclear, actually, whether in debug mode, we normally care about machines coming up. Maybe we do. Actually, let's do this. So instance ready. And then we want this to include its set, which is going to be name. And it's IP, IP, probably IP, public IP down here is when we establish all the SSH sessions. So we probably down here want connected to instance. This is guys, I'm looking for cheap laptop for programming. I will let the people in the chat help you rather than take time. But I'm sure there are people here with recommendations. Right. So here, this is after we've successfully established SSH connection to a machine. Arguably, the fact that an instance is ready is a trace level. The fact that it's about to be set up as a debugging event. I think I want the name to be in here. And then I want a tag that says IP.  Check the still compiles. That seems like a nest log bug. It's really annoying to have to keep adding all these. Great. Okay. And then I guess. Oh, that's a good question. I think this is maybe our first info. There's an argument for this being an info. So remember info is printed even in release mode. No, I think in fact maybe this is that's a trace. And this is an info or this is a this is a debug. This is an info. And then benchmark running main, what are we calling the the thing that you run experiment routine. I guess routine is like a main sound is very fancy. Or actually, how about just spinning up tsunami? No, that's sort of spinning up the machines. What do you do with a tsunami? It's unclear how this how this naming is going to help us because arguably that should be the first thing it prints spinning up tsunami. It's going to be info number one. And then down here, I guess we could have an info down here somewhere that says info log all machines up and ready all machines running set of routine running setup. What is this going to be though? Tannami is triggered. Yeah, although we're OIC so we like spin it up in the beginning and then we trigger it. It could also be interesting wrecking havoc. But it seems seems unfortunate. Unleashing the power of the tsunami. That's good. I like it. Force wave. That's also good. No, I like unleashing the power. Sounds very satisfying. And then I guess tsunami is no. Oh, wait. The alternative here is to make this quiet before the storm. Power of the tsunami unleashed. The power of the tsunami was unleashed. I like it. And then I guess those should both be info level events and then debug. Actually, this debug is a little bit unnecessary, but sure. Tearing down instances. I guess terminating. Terminating. I also kind of want this to tell me how long it took. Do we have time? Let's see. So and then release the kraken is also pretty great. Releasing the kraken. It's not entirely related to a tsunami though. Oh, yeah. Fun log messages are definitely the goal of any library. Oh, so this is an important point, though, that one thing that's really nice about Slog is that it doesn't actually log anything by default, right? So the way we've constructed our library now is someone else can use our library as a library in their library. And then they can layer logging on top of us, right? So you can imagine they give us like a sub logger of themselves that contains some tags. And we don't have to change our code for that. Our code will just integrate with however they want to do logging. And I think that's a really sort of powerful composition that we can do there. I really want a time pretty printer. But let's just do maybe I'm going to assume that this is going to take multiple seconds. And then this is going to be that's going to be a trace event retrying instance termination. Okay, so this is a case I don't know if you remember from the previous stream. This case is when you if you haven't interacted with the AWS API for a while, you're like session expires. And then the way Risotto exposes this is not that it retries internally is that it gives us an error. And it's a string errors, we need to like do string matching on it. And look for whether it was disconnected or had a broken pipe. And if it did, then we retry, which is fairly silly. I guess technically, this should now be here. We should be a debug cleaning up resources, temporary resources. And there should be a trace cleaning up security groups. And here, this should be a debug of, let's see if this compiles. So close. 379. Did not ask sex. Oh, a lapse. Let's see what happens. Connection errors are the hyper connection pool, not correctly marking dropped connections as unusable. Oh, I see. So in the newer risotto, I see in the newer hyper, which is used in the newer risotto, this problem is fixed. So I think there's an underlying issue too of I want the, the library that we're using to expose slightly more structured errors. And it's not really doing that at the moment. But at least if I don't have to do that retrying, that helps a lot. Because there are transient AWS errors too. You plan to write tests for this live. Okay, so that's a good question. I will show you. I really want to write tests for this, but it's currently a little bit annoying because of this. I really want this bike to be fixed. I've had some productive productive discussion with one of maintainers, but maybe that's the person that's here. I'm clear. So I really want to be able to mock risotto, and especially the APIs that I'm using. But it's a little bit inconvenient at the moment. And it would be a lot of work to get that to work well. Writing unit tests might be a little bit easier. But the problem is basically all of this crate is interacting with Yeah, most of this great is just using the AWS API. So without being able to mock that nicely, there isn't even that much to unit test. Like our code, there's not that much code that we have that does useful things. Yeah, I figured, okay, so we do have one of the, we have this guy's in the chat for those who are not observing the chat. But yeah, I think being able to do some kind of mocking, it doesn't necessarily have to be this way, but just having some way of mocking the API would be really useful because that way I could actually write tests for this. Um, this seems pretty promising. Connected easy to spinning up to Nami, using spot requests, waiting for them to spawn, all spawned, then all the setups, quite before the storm, ran the thing. The tsunami ran in zero seconds, which seems fine, because it currently not really doing anything. Terminating this isn't all done. Seems like a lot of really nice logging. So, okay, so the thing that we now want to do just to show you the really neat thing you can do, so notice that we're no trace events here. And in fact, they aren't even in the generated code, the compiled code. So what you do is you do this. Hopefully this would be commented out. And then we just not come in this and notice that this will go. So if this is combed it out, then what happens is that in debug mode, everything only things above trace will be shown. In release mode, only things above debug will be shown. If I remove this commenting, then now trace level events is shown in debug mode. And debug level events are shown in release mode. So let's try that now. Did I do something stupid? Probably, but oh, let's see. So in theory now, because we're compiling in debug mode, we're not using dash dash release, we should also see all the trace events. So notice how this kind of output would be extremely useful if you were debugging and trying to figure out why this doesn't work. That seems pretty nice. I like it. And notice here how the tags that we added are just like added as like key colon value pairs after each line. Whereas in a more structured logging output, then you would get them as separate fields. So for example, if you ask for JSON output, then now those keys would be easily available. This is where it tries to do the SSH connection. Remember, it also has to wait for the instances to reboot. So it's not entirely outrageous that it takes a while to start. We might want to trace connecting to them. Basically, now we're just like, keep connecting until the machine is up and its SSH server is started. And so it sort of makes sense for this to take a little while. Oh, that's interesting. Yeah, so notice here that it didn't run the main routine, which is sort of what we expected to do, right? And so it failed setting up one of the server instances. Arguably, we want that log in error of some kind, like up here. I think what I want is dot map error. I guess, actually, this is error. Yeah, specifically, this didn't actually print any errors in our logs, which is a little bit sad. And it's because it like returns the error that happened immediately. So it's unclear that we actually need to log that you could imagine this being logged by the higher level. But specifically, like failing test is H2O machine, I think is sufficient. Or maybe it's not. It's a little bit unclear. But I think I want that to be logged as an error. Arguably, like the main routine failing. Yeah, let's do that too. First of all, this no longer needs to be a map error. So here as well, this is actually critical, right? This is the main loop fail to run, which would make us really sad. Main routine. I think those are like, I guess this as well. First of all, this map error is no longer needed. But also this should map error. And here as well, we want a error set up for, yeah, so in theory, that basically, so the question is like, are there any warnings? And I don't think our current setup needs warnings. I don't know exactly what those would be in our case. It's basically the errors are you failed to as H2O machine, you failed to set it up. And a critical error of you failed to run the main routine. The fact that it failed to connect here is a little bit disconcerting. It suggests that our ssh time is not long enough. And I guess the question is like, how long does it take to boot a VM? That's really what this time is. Arguably, it should be configurable. But for now, let's make it longer. Should not take more than two minutes. Just run that in the background. I think that gets us pretty close on logging. Right? There are any obvious things that I've missed that you think should be logged? Because like this seems pretty useful. And again, you can the trace events will not normally be there. That's just because I've in our cargo dot com will have this feature enabled, right? So if we remove that, all those trace logging statements would go away. Instance start time can have a real long tail. Yeah, I mean, I'm aware of this that AWS sometimes takes a really long time to spin up instances. The thing is, it's a little bit awkward, right? Because the, in a sense, the right thing to do would be if you fail to connect, then check that the instance status is still active. Because you don't want to keep retrying if the instance has failed. But that would sort of break an abstraction boundary for us between this SSH session connection and the setup stuff. But it's a good point. Yeah, I wonder whether it's added to do for that here. So instead of max time, keep trying as long as instance is still active. I think that's what we want, right? Could cargo features be used to configure S log level state cargo run features debug to get trace level logging? Yeah, so one thing that I could totally do is to have this features. That's actually a good question. Okay, so the question was, can I have something like debug here, which would be a feature that when enabled turns on like s log dot max level trace, right? That's basically what you're asking. I don't think you can enable sub create features this way. Okay, yeah, so this now did the right thing. Great. I think I'm happy with that. That's a good question. Let's look at the manifest for docs, cargo Tom features, features, um, feature groups can only reference optional dependencies. Yeah, it doesn't look like you can. That's a little bit sad. Although maybe one way to do it would be let's see. So for dependencies, can I rename a dependency? Because if so, you could totally do this, right? So you could do like you could imagine doing something like, right? So that's something that maybe could work, although you would have to have a way to say don't include the normal s log. But yeah, it doesn't look like you can do this cargo feature. So no, unfortunately, it doesn't look like it. Although remember, one way you could do this is you could have, if someone used my crate, they could do dependencies dot override, isn't that what it's called? cargo dependency, actually one patch, I think I want to replace. Yeah, so there's like, I don't remember exactly the syntax here. But I think they could do something like override tsunami s log with like, if I remember correctly, you could do something like this. A default optional dependency of s log an optional feature that imports s log with those other settings. It's a little convoluted. Yeah, although I feel like there really ought to be a way to do this. I'm sure there's a cargo issue somewhere. Yeah, features cannot enable disable features of dependencies closed here. And it was with the bar dependency in the best feature of bar. Yeah. So in fact, we should be able to do if I read that post correctly, features debug is equal to s log slash max level trace. Maybe I mean, who knows, but features that sort of seems like maybe it does the right thing. Maybe. Although can I enable multiple features? It's a little bit unclear. Yeah, so this gives us trace with features debug. In fact, maybe it can even do s log slash release max level debug. What do we think? Yeah, so that works. That's not particularly well documented. But sure. Great. Well, we learned something new today. I did not know this. Great. All right. Now I guess I wonder whether you can enable multiple features this way. But yeah, that's pretty neat. Okay, so I think that mostly brings us up to speed on logging. So I think we can do this. I think scope guard is what I want to do next. It's going to make clean up a lot nicer. Great. Okay, so if I now do check. So there is a requirement that features are additive, which I found recently. So sort of seems like this should do the right thing. Which means I should be able to do this again. Maybe. But that didn't recompile s log, even though the features should have changed. Something's weird. I don't like it. Right. Let's just leave that. Actually, how about we do this? Would this be used to multiple? See, we're contributing. Good job team. What comes after buzz? That's a good question. Foo bar buzz, then what? Foo bar buzz. Medicine tactic variable. What comes after that one? Foo bar buzz. Quicks. Okay, let's use that one. Learn two new things today. Great. We're about to find out. Let's just leave this one for now like that and then do this. And then leave a note for ourselves like this. All of this is logging, right? Almost. So close to you. Not this to do. Yes, logging. Great.  What else? Give us more time. Right. Now we're going to deal with scope guard, which is another really cool crate. So scope guard. So scope guard is a way to have something run when it goes out of scope. So one of the issues that our current code has is that imagine that you crash somewhere in here. We're not necessarily crash, but like we issue the spot requests and then terminating the spot request fails for whatever reason, or maybe a better example is terminating them succeeds, but then we fail inside this loop. So we return an error. Now nothing will terminate the instances that are currently running. Because like remember, the termination happens down here after we run the main loop. So one way we could do this is like, if this thing returns an error, then we go into the stuff below, which actually isn't terrible. But the alternative is to have the termination code and the cleanup code be inside of a scope guard. Because that way, whenever that goes out of scope, whether it's because of a panic or return, or whether it's because we get to the bottom of this scope, then the scope guard will be run and the termination will happen. So basically, instead of doing this all up here, we would do this. So we're going to make some like foo thing here that's going to just like sort of sit there and remember how to do tear down and cleanup. And that's exactly what scope guard gives us. So scope guard, what version is it? 033. So we're going to have scope guard is equal to 033. So we had a scope guard. And now down here, if you look at scope guard, there are a couple of different ways to set it up. scope guard, I think we can just use defer. So we're going to need to macro use it. Macro introduces a macro. And then we defer this. Let's just like see if that works naively. I don't think it will, but we'll see. So you get the intention here, right? The intention is this is similar to goes defer keyword of now that we know the machines have started defer tearing them all down. There's still this problem of it could be that we fail in this loop, right? So we describe the instances, and then some of them come up and some of them do not. In fact, we might have to deal with that case somehow. The problem is, imagine the describe instances fails for whatever reason, there could still be instances that are running. Little bit unclear how we do this. I think this specifically has to happen before we even describe them. It has to happen. And we have the instance IDs of all the machines, like, like where the to do is that says this is where we'd create the scope guard, for example. Yeah, so this is here, all of the machines, all of the spot requests have been satisfied with instances. So there are instances running. Now we want to do some cleanup, let's do deferred cleanup, so that when all the other stuff ends, whether it's because it returns early, or whether it's because it finishes, then we want to terminate all those instances, and potentially do our other cleanup. In our case, because this is commented out, this will hopefully be fairly easy. There's sort of a weird condition here of stopping the spot requests. Because if this fails, we really want to try to do it again. Because if, in fact, maybe we just need to, it's a little bit tricky, because like us, the way an Amazon spot request works is it will keep, it will try to ensure that you have as many instances as you requested at all times. So as long as the spot request is active, Amazon is just going to keep spawning more instances if any of them go down. So if we terminate them in this defer, then the instances are just going to come back because the spot request is still active. So that would be specifically if this thing failed. So this thing failing is actually pretty bad for us. But I don't see why it would fail. Like if there was a transient error here, it would have some serious issues, but this is a little better than nothing. So did this compile? Probably not. 346. Yeah, so one of the, oh, what? 346. That's a lie. That's not the right line. Oh, I moved things. That's right. 285. Yeah, so here, all right, the instance, what we really want here are the instances. So instances, the problem here, of course, is the scope garden needs to have its own copy of all the instances. It can't, the instances array, the thing that keeps track of all the instance IDs that we log up here when all the spot requests are satisfied, those are currently sort of carried around all these places. Like in this loop, we keep, what's this here? We keep trying to describe all those instances, so that value is sort of used elsewhere in the code. So we can't move it into our defer into our scope guard. Instead, we need to keep a copy of it that we're going to use in the scope guard. So this is what we're doing here. This is going to be termed instances. And then the other error we get here, of course, is that in a defer, we can't really return an error, because there's nowhere to return it to like imagine this function is already returning, we can't then like have it do like, now return another error. That's not a thing we can do. So we do want this to the logging statements still be here. But if it fails, we can't really do this, we can't return. I think we would just have to do warn log. Maybe this is a good news for warn. Yeah, so basically, if we failed to tear down the instances after something happened, all we can really do is warn the user that there are instances that were not terminated. Arguably, we should do the same thing with the spot requests here. So this is going to be warn log fail to cancel spot instance, spot instance request. Let's see how that works. Okay, so this is a, so what it's complaining about is it's saying I'm not allowed to move term instances here. The reason for this is that scope guard is that drop does not let you move out of self. So if you if you remember the drop trait and rust is only given a mute self.  And in the case of the scope guard, the mute is the environment the closure is running in. And this means that it can't consume its environment, which is term instances. So I think the thing to do here is just split, split off zero. This is a neat trick for if you want to take an entire vector, you can use. So split off returns. That's not actually what I want. In fact, let's do it the other way around. So we can use standard mem. mem has a method called replace. replace takes a mute of a mute t or rep mute t to something and a t and swaps the two and gives you back the one that was there. So if we might be easier to explain if we look at it. So mem replace takes a ref mute t and a t and gives you back the t that was in destination and makes destination now point to this t. This is really useful for vectors because you can do right. So now term instances is pointing to an empty vector, which doesn't do any allocation. And we get back the actual term instances. I feel like that has a method for this, but I may be wrong like split off would sort of does the reverse when given that we want all the elements could self would contain all the elements, but we can't actually get to self off the top of my head. I can't remember. Anyway, okay, so mem replace should do that. What else is it complaining about? Oh, right. Hey, boss. Okay, so in theory, now there should just like never be any instances that are left over. That makes me pretty happy. So remember, this is one of those things where our API hasn't really changed, but the behavior of the API is like much more pleasant now. Now it won't just like leave lots of stuff around. And so if you were to run run this now, I mean, we should just to check, but this should not really have changed anything. And in theory, our to do should no longer have scope. I think given the time, I think what we'll do next is we'll do documentation and like cargo tumble stuff just to tidy up and sort of tie and I need bow around what we have so far. And then async IO and stuff we might do later, unless it's something in particular people want to see. I think documentation and cargo tumble is probably the way we'll go for the rest of today. Yeah, so notice now without the dash features debug flag, just like we wanted it does not print any trace level events. Yeah. And so see how it was all done. And then it prints terminating instances afterwards. And this is because if you look at it, all done is printed like at the end of our loop, the next thing we do is return. And then the scope card is dropped. And when the scope card is dropped, that's when it runs this which then prints terminating instances. So this is exactly the kind of behavior we would have expected. And if there were to be an error, then this would still be run afterwards, which is precisely what we wanted to want it to happen. Okay, so this means that we now have four almost guaranteed. That makes me very happy. Let's see. I really want to fix this. But so I'm oops, I may end up fixing some of the smaller thing. Oh, the nicer SSH interface would be good too. Also, I want someone suggested this last time of having a neat way to be able to collect performance results from all the machines you ran things on, like you can imagine having the having like, at least that you time or collect CPU resources from all the machines over the course of the experiment, and then you return those in some nice format, I don't know exactly how that would be that would sort of be an API design type thing. But let's yeah, I might end up like fixing this offline at some point, because it's just a little bit annoying, but it's not really that interesting. So it's probably not worth spending time on documentation though. Okay, so for documentation, what I really like to do and usually the first thing I do with any crate, I just didn't think it would be productive in this case, is that I add the following to my time. This is the first thing I add whenever I write a new library normally. What this does is if you try to compile, if you do not have comments for a public thing, then it will not compile. And this is a good way to like force yourself to actually write the docs, you can still write terrible docs, you should probably not do that, but you could. But at least this incentivizes you to not leave things out. So let's see, let's start with fine, let's start at the top, let's find why do we expose machine? We expose machine as a handle to a machine that has been spun up. So handle to a currently running tsunami to an instance currently running as part of a tsunami. Sounds very dramatic. It really also bothers me that this has to be an option. This is another like ergonomics thing we might want to fix to this machine. I guess this should be host. AWS EC2 instance type hosting this machine. Host instance, the machine are like, they're not good synonyms. AWS instances. One thing that is really nice is in your docs, like using more words is fine. And in general, try to be helpful. Like in this case, where is it here? We specifically want to list link to this. Is there a way to do that? Does this have its own like URL somehow? Not really. That's kind of sad. I guess actually that's not the page they should see. They should see this page. Going to be used elsewhere. Private IP address of this host on its designated PC, publicly accessible IP address. So this is one of those things t