Yo, we're IRL. Sorry, I'm so late today. Chad, we're trying. What's up? We have bombies here, venomous here. So I didn't even know this was going on, but there's like a convention in Austin, GalaxyCon. It's like, is that going to animate, sci-fi?  Which I wasn't ready for, but actually it's just an anime convention. Yeah, yeah. Sick. It's so fun that conventions are like coming back, Chad. Oh, dude. Look at the pyramid heads from Silent Hill. That looks awesome. Can I show it? Can I show it on camera? Yeah. Look, is this incredible, Chad? It's so good. Contest is seven? Nice. Yeah, that is awesome. Yeah, I love Silent Hill. That is so sick. Hell yeah. All right, good luck. Ooh. Wait, this place is actually crazy. Yeah, there's a ton of stuff out here. Dude, my chat thinks like a, I'm not like super into anime. I like Dragon Ball Z, and I'm like, I'm broke. I think like the show is like, it's cool, right? I'm not into anime. But yeah, I'm not.  Yeah, I'm not a weeb, but I don't hate them. But I think I'm a weeb. I'm like, I'm not a weeb. Do you mean like, meet the slur or like, meet the comic? Both. Whenever you cosplay as something from a series, and then you see someone else from your series, and y'all got to do it yourself. Oh, true.  True, actually. For example, if you're Frozone from Incredibles, and you see like anyone else from... Yeah, like Mr. and Gretel or something. You got to get the, you got to get the... 100%. What did you get? She was off camera for like 40 seconds. She was buying stuff. Try P2W. Yeah. I'll walk right out. If I ever buy things at conventions, I usually do it at the van. Because I don't want to walk around with stuff. Yeah, this is why I say whenever you're talking to them. A great, a great finisher is, let me finish my first lap and I'll come back around. You know what I say? It's like, okay, I know who you are. Now, I'll delete the vod. If you don't give me stuff for free, I hold them hostage. Oh my God. This dude. Not that's what I used to do years ago, but what I used to do years ago, I would go to like conventions and I'm like, okay, here's the goal. I want to farm as many free things as I possibly can. Oh no, Sean. Look at these weapons, dude. Look at these. Look at all these weapons and of course, a hand of rag. No, these swords are small. Dude, a hand of rag. I wish it was bigger, but yeah. That's like the, basically like the first legendary in WoW. But it's funny because it's kind of small in-game too. Everyone's like, why is it? Hand of rag is kind of small, but dude, that's crazy. Like you got master swords and this one is from WoW. That one up there is from WoW that's, so that's a gore howl that acts. The art for like the old WoW weapons were so cool. Now they all look like laser beams, I don't know. Yeah, it's called a gore howl. Yeah, it's foam, Chad, it's like foam. But it's like, I mean, it's good. Oh dude, look at this guy's sand. Sand armor. Sick and the scouter, nice. Oh yeah, you're good. No, no, you're good. We're looking at you. Yeah, oh yeah.  Oh, dragon ball scouter. Oh, nice. I think he wants the dragon balls. Dragon ball, bring it on. Sick. And that's the whole set. We have the single one right over there as well, you guys. Single ones? They're a little bit bigger. How much is this? This one's 65. Okay, like it's kind of weird, but like when I was younger, like I didn't think anything of Sailor Moon, but now as an adult, I'm like, you know, Sailor Moon is kind of weird. Cause, sick. Oh wait, yeah. Dude, we saw one of those that like that big thing that me, Miz and Ludwig did. Yeah, like the switch fork, what do we call it? It's not actually a switch fork, that's what we call it. Yeah, cause they're like high schoolers, right? Yeah, well for me it was like, I thought Sailor Moon was cooler than Dragon Ball at first. I would pretend to watch Dragon Ball. My dad would be mad at me for watching it with a girl. A girl show? Buy the hand of rag? I kind of do want to buy the hand of rag. If I have like all my actual like encyclopedia of WoW weapons that I've accrued in game in real life, for the complete and total character vision. Whole keyblades actually. Oh my gosh, the keyblades. These look great. Hi, how's it going? I think if I got a keyblade, I would just want like a baseline. Nah, I want the drip, bro. Really? I want this, the Woody Buzz one. Sorry. Hello, it's me, Mario. We've been looking for you. Really? Yeah. Well, I am here. We found you, we found you. Yes. Great job, man. Thank you very much. Yahoo! Yes! Super soft. That's a really cool way. That's it. Dude, he was so good at the voice. That was crazy. He was actually perfect. It caught me off guard. Yeah, I think that was actually Mario. I think that was him. You have like mounts for sale? I got like metal, like swords. Oh, like general weapon mounts. Oh, like that, okay. But do you have anything for the wall? And a wall mount? You know what, now that you're saying something, I should start looking to get something for like a wall mount or something like that. Yeah, I think if you had a wall mount, I think people would love that. I think you're right. I think I'd be sick. You know what, I'm gonna start looking out for that. There you go. Cool. I'll tell you what, I get so many different ideas from all my guys. Yeah. We're always constantly evolving, you know. You guys have a website? Yeah, you know what, here it is. Your Chad, if you wanna check out their stuff, Triforce Cosplay and accessories. Awesome, thank you. Nice, I appreciate it. I like when people give me their card and I like show their card on camera. But if people in Chad who end up buying stuff because they're like, oh, that's sick, but like they can't actually go to the conventions and then they end up buying stuff online. Everybody wants Kingdom Hearts too. It's being a Kingdom Hearts. Dude, look at all these controllers. Dude, old consoles. Dude, 3DS has like jumped off now. They're so... Really? Yeah, like 3DS is like what could be plus now. Really? These are actually good prices for 3DS. Really? Dude, I remember this guy. Yeah, I actually never had one of those. I never had an original Xbox. You wanna feel weird? 17 years old, right? That's wild. Is that weird to think? You know what else is wild? The fact that Switch has been around like, I think seven years. Yeah. Seven years in March. Yeah, there you go. Almost seven years. Are these like mystery boxes? Yeah, I think they're actually like loot boxes. I've never bought one, but isn't it kind of interesting that like, loot boxes are so successful that they started doing loot boxes in real life? They have the office. Wait, that's actually... The office, the Star Wars. The heck? Gamble has invaded IRL. Dude, that is wild. Dude, the normies invented me. Yeah, right. Aware, dude. It's so interesting that there's so many different things that I would never think about, like the office. What can you get out of like, a box for the office, for example? So, the office things. Yeah, well, I am so hungry. And I know, and it's a trap. It's a trap because convention food is mass price couch. Yeah. But, god damn, it's so trash, it's so trash. It's quite, it's quite sensational. So the thing is, when you're hungry, when you're hungry, anything tastes good. This is sick. Nice. Perfect. People haven't been doing that all day. Just finding people with cameras and then just turning and looking at them. Hey, how's it going?  What's up, man, how are you? I didn't even know you were gonna be here. Me neither. I didn't even know. I've never met you in person. I've had you like, I'm in the background of my office for like, four years. Sick. Oh yeah. Anyway, it's nice to meet you, man. Yeah, of course. You gotta get like, I saw, here's something. No, absolutely, yeah, let's do it. Oh, or whatever. No, let's do it. Awesome, dude. Thank you so much. Yeah, of course, man. Yeah, I'll see you on stream. Yeah, I'll see you. Hey, how's it going? How you been? Long time no see, man. It's a good day, man. Yeah. Are these candles? Yeah, these are still, my wife and daughter made these. Oh, you've got mocha flakes, cinnamon, pumpkin. Yeah. Do some strawberry jacqueries. Those are cherry Coke as well. Nice. Do you have like a website or anything? Yeah, right here. Great, if you go to these two right here, you can pull up their information. And yeah, they have all the stuff about where they're gonna be at. Where they're probably gonna be doing Giganticon pretty soon. Cool, I don't know if the glare matters, but. Yeah, I remember we met at, it was Dreamhack 2810. Yeah. I'm sure I got to see another picture on there. One of our friends was like, you got pictures with us? Yeah, yeah. Yeah. Aswin at the same time? Yeah, it was me and Aswin. Summer was there? The whole crew, yeah, a whole bunch of people were there. Chance was there. Doctor Disrespect was there too, as I mentioned. I did not know he was that tall. Yeah. I knew he was tall, but that was insane. Yeah. What's your biggest plan for? So I start tailgating on Saturday. I don't know where I'm going to, I don't know what school I'm going to yet, but I know it's starting on Saturday. I'll either go live at 12 or three, one or the other.  Yeah, we're like trying to run around because I have to go kind of soonish. But so that might be trying to drive me. But yeah, it's good to see you, man. Yeah, this is great. Yeah, these are super cool. Yeah, we'll see you around. That guy, that guy's like an old school, like, viewer of mine, like from like six years ago. Hey. There are so many things to find. I'm freaking out. Yeah, you keep running away. We're like, where are you? Oh, dude, this is great. This is so good. This looks so safe. Yeah, I'm so happy. Yeah. That's so good. All pencil. All pencil. Got the Woody and the Buzz. Do you have a, do you have a card? Do. Oh. There you go, sketching time. Some time productions. That's me, Tracy Watkins, sketching time productions. There you go. Thank you so much. Have a nice day. Thank you, you too. How are you doing? I'm good. Good, good. Oh, you guys got camera duty? Yeah, yeah, I'm camera-manning for him today. Awesome. Yeah. Thanks for your hard work. Yeah, of course. Thank you. I appreciate it. Yeah, I'm trying. I'm trying to do my best. Are you getting any merch? I'm keeping track of all the, like, different websites and stuff, so if I want to order anything or get something at the end when we're done camera-manning. Yeah. Do you have to get, like, all the buttons and the keychains, or do you, like, just the apparel? I like stuff that I can put, like, up in the house. Like, if I can put, like, a little, like, weapon or something. Absolutely. Yeah. Yeah. It's so good. Yeah. For sure. Camera and talking, my bad, my bad, my bad. Wait, oh, that's too bad. Dude, look at, like, the shape of my face changes. Isn't that crazy? Yeah, of course. He spotted you for a mile away. Oh, yeah, really? It's probably the hair. Yeah, of course, man. A birthday present for me. Oh, watch out. Okay, I'm closed. Okay, my eyes are closed. Okay. I don't want that. Ew, Jesus. Why does that, why does that exist? Can you give me, like, a, like, a paladin? Hey, how's it going, dude? What's up? I knew you were awesome. I'm here. Bruh, bruh, bruh. How's that, man? Bruh, bruh, bruh. What's the problem?  This is a lot of scuttering, but you don't like, like, well, no, I'm just saying, it's hard to explain it, but, like, I want to see, like, a paladin. Not, like, booga. You know what I mean? Because what if my mom walks in?  Yeah, a big, big sword, big titties, and a big, big, other ties, and yeah, I agree, yeah. Like, why is everything, why is there to be, like, a booba version of everything? Like, it's a Pokemon. We're in Texas, everything's great, right? What do you think we want? True. Are you a communist? They have communists. Oh, dude, look at this dark-wing duck. Look at that, this is so cool in the Robocop. It's super cool. Oh, Corellitive Hill. Super cool. I'm, like, obsessed with it. Yeah, I just randomly found it out, and I was like, wait a second, this is sick. And I'm going to be doing this forever. Oh, nice, look at that. Chad, this is what SpongeBob looks like after that one TTS. Yeah, SpongeBob. Look at this one. Dude, I love shadow boxes. I bought a few of these before, I just love them. I need to mount them. Dude, yeah, I need to mount them on the walls. That's what I need. But I absolutely love shadow boxes. Oh, the, oh my gosh, it looks so cool. Dude, that is sick. They're light, not shadow boxes, they're light boxes. They're light boxes, they're the gallery pendant. Dude, I absolutely love looking at this stuff, dude. Hey. Okay, pose. Okay, why is this the same? Swords, swords are, oh, you can tell. Dude, that, like, look at these. Aerith, Sephiroth. This jinx is cool. I like these a lot. You like those? Oh my, what the frick? Is that what you wanted? No, I thought that. That's what you wanted over there? No, that's not what I meant when I ran it. The Pikachu? No, I want, like, look at this guy's got a big sword.  Yeah, he does. I kind of want that hand of rag. Oh, dude, this looks sick. Dude, you're owning. What the heck? This is awesome. Look at that. The helmet too. Do you make everything yourself? Yeah, I'm not the helmet's black series, but these are two-in-connet. Oh, nice. This I found on Amazon, as well as the shoes. Nice. And the suit underneath. Nice. Sick, dude. You're owning. Good job. Yeah. Hey, how are you? What's your name? Yeah, I'm Esme. Yeah, good to meet you. Yeah, okay. I didn't register for it. Yeah, I mean, did you meet me? Yeah. I was like, wait a minute. Yeah. You know him? Years ago, yeah. Oh, okay. Cool. Yeah, years ago. Like that. I haven't been streaming stuff for a while. Yeah. Yeah, so I don't say that. Cool. Sweet, man. Yeah. Yeah, I'll see you. Here we go. That's money. Oh, somebody bought the hand of rag. They replaced it with a pickaxe. Somebody bought it. No way. I was actually thinking about that. I was just like, it's a little small. So, oh, you got it? Nice. Nice. Cool. Hi, birthday. It's for me. Sick. It's for you. That's like what I did when I was little. Sick. Oh, sorry. Sick. Dude. Dude. Dude. Dude. This is my, this is the first legendary thing that I had in my life, wasn't it? That's sick. Well, I mean, Legion doesn't count. I came back for Legion. I'm just like, randomly doing it. But, you know what? It's actually probably, even though I was like, it's not that big. I was like, in wow. It probably was about this size. Sick. Thank you. Hey, how's it going? Nice. What's up, man? How's it going? I got my hand of rag in game and in real life. Sitting on my ballad. Oh, look who came back. Hey. Lucky man. Yeah, I got this now. I got loot. She's really hooked up on that one. Yeah. Sick. Cool. Thank you. You're welcome. Yeah. All right. Have a nice day. Yeah. I guess this is what I have. Sick. Thanks, Bonnie. Can you carry it? Yeah, I'll carry it. Sick. Cool. All right. We'll see you. We'll see you guys later. Bye, everybody. Hey, make sure to turn on your notifications for stream. Every Saturday, IRLs, events, shows, whatever. Go follow everybody. Go follow Venom, Trihex, Bonnie. Thanks for everything. I really appreciate it, guys. Thank you so much. And we'll see you guys later. All right. Bye, Jack.


Ukraine liberates two more settlements in Zaporizhia. Fierce battles are raging in the direction of occupied Melitopol and Veslevka cities of the Zaporizhia region of Ukraine. Report Agency informs. The Agency filmed one of the combat operations during the counterattack. As part of the operations, the special units of the Ukrainian Army destroyed the Buratino artillery unit and several grad systems of the Russian Army. Alexander Thoran, the senior officer in the Ukrainian Armed Forces, told a report that during the last few days, units of the Armed Forces of Ukraine liberated two villages. Currently, we are fighting for three more villages. Morale is high in the army every day we fight and move forward. Our task for today was to destroy the Buratino device. We waited for it to come to a combat position and destroyed it with eight missiles. Alexander Thoran also thanked Azerbaijan people for their support.


All right, so today we're going to talk about how to performance. This talk is going to focus on tools and techniques to help speed up your Ruby code. The things we're going to talk about today came out of work that I did to make integration tests faster. I'm Eileen Yuchitel, and I'm a programmer at Basecamp, formerly known as 37 Signals. I'm on the security and performance team, which means I make sure that things run fast and smoothly, safely as well. You can find me anywhere online at Eileen codes. I'm also on the Rails committers team, which means I'm allowed to push directly to master and on the Rails security team. I'm not on Rails core, which seems to mean that I'm allowed to tweet from the Twitter account. I'm not sure if that makes you feel more or less secure. This is my dog, Aria. She is named after Aria Stark from Game of Thrones. She has her own Twitter account at AriaDog, and she'd love it if you followed her. Be warned though, she likes to talk about poop a lot. I'm from upstate New York from a city called Kingston. It's about two hours north of New York City. It was the first capital of New York in 1777, but then the British burned it down twice, so they moved it to Albany because we can't be trusted with it. A fun fact, I'm not actually the only speaker from Kingston, New York. Aaron Quint literally lives on the same street as me, and he's speaking after lunch, so. Kingston's representin' hard. I'm really happy to be speaking at Fullstack Fest this year. This is actually the first European conference that I've spoken at. Barcelona's a lovely city, and when I told my coworkers that I was going to speak at Fullstack, they asked me if I was going to be talking about pancakes. They were pretty disappointed that I was not. And good thing about before lunch, so that you can all be really hungry after looking at this picture. So before we can talk about speeding up integration tests, we need to go over the difference between benchmarking and profiling. Benchmarking measures how slow our code is. This is all about collecting data on where your code currently stands. Benchmarking tools are also instrumental in proving that you made your code faster. And profiling tells us what parts of our code are slow. We can use profiling to pinpoint the exact part of our code that's slow. Improving performance is a lot harder, if not impossible, if you don't use these types of tools together. So let's get started. Back in January, DHH said to me, hey, I like fast things like race cars, and I really hate it when things are slow. I want to move BaseCAN's controller tests to integration tests, but right now they're 20% to 40% slower than controller tests, and that's just not going to do. Can you see what you can do about speeding up integration tests so they're as fast as race cars? Of course I said yes, because this is a good opportunity to improve for everybody. If we're going to improve performance, what's the first thing that we have to do? We need to have a baseline. This gives us something to measure against so that we know that we're actually improving our code speed instead of just thinking or hoping that we are. To get a good baseline, I decided it would be a lot easier to not change the BaseCAN code and rather build an application specifically for testing integration test speed. Building an application has a few benefits. It's public and it's on GitHub, so all of you can download it and play with it and see what we did to make them faster. And it removes anything that can be getting in the way of helping me build that baseline, like setup code, authentication, copy-bearer and all that crazy stuff. Of course a bare test, like a Hello World example, isn't going to show you where everything is slow, but you have to start with the bare minimum and build up from there. Controller tests and integration tests don't look that different. Both define the test, process a route and assert something about the response. Here the index tests for both are getting the index and asserting the response was a success. To build a baseline, I started Ruby 215 and Rails 420 because that's what BaseCamp was using at the time. I wanted to be sure integration tests were slow and isolation using the same technology as BaseCamp. First I double checked DHH's assertion that integration tests were 20 to 40% slower than controller tests was actually true. He was basing this off the time they took to execute, so I first looked at that knowing it wasn't going to be my real baseline. I ran the single controller tests that we looked at earlier and found that it took 1.625 seconds of real time. I then ran the corresponding integration tests which took 2.12 seconds of real time. This calculated to integration tests being about 23% slower than controller tests. It's actually a huge difference. It would make running your entire test suite on integration tests quite a lot slower. I decided to run each of the tests five more times and got sort of interesting results. The time a single test took to run varied a lot and after a few runs, it looks like there's no slowdown at all and at the same speed, but I knew that this wasn't true. It points to the fact that time is not a good baseline for this type of performance work. It's good for getting the initial is it slower but it's not great for actually benchmarking.  To use time, we'd have to calculate the average of all the runs and also figure out how many runs we need to do to get a good average. Time wasn't a good baseline because it varies a lot and it would have required a lot more work than I was willing to do to prove it. I was making it faster. So if time isn't a good baseline, how do we get a baseline? We can't start improving our code without one because we don't know where we currently stand. A good gem for benchmarking your Ruby code is benchmark IPs. It was written by Evan Phoenix and the gem measures the number of times the specified code can run in one second. This means that the more iterations, the faster your code is. It completely takes the guesswork out of figuring out, there's like notes for me on the screen telling me to move over, but then I can't see my notes. So that's not going to work. Sorry. So benchmark IPs means that the number you can figure out like your code is faster because you can run more iterations. So you can run a block of code like 20 times versus five times. It really takes the guesswork out of figuring out the number of runs it takes to get a good average. It also provides the percent of standard deviation so that you know how far spread out your results are from that average. Benchmark IPs allow you to focus on the results rather than the data points when benchmarking your code. Here's an example of the benchmark IP script I use to measure the difference between integration test and a controller test. First we define the controller test. It's a single get, a simple get with a, and we assert the response was a success. Then we do the same for the integration test, get the index route and assert the response was a success. Then invoke the benchmark IPs gem. Inside the benchmark block, we report on each of the tests. Using many tests, run one method. We can call each of the tests inside the code and label them integration test and the controller test so they're easy to recognize. And then we use the compare bang method from the benchmark IPs gem to compare how many times slower integration tests are than controller tests. This method is great because it does the comparison math for you. We can then run this like any other test and it tells us that we were able to get 393 iterations of integration test, give or take 7%, 8%. And that we were able to run 920 controller tests and give or take 7% in the same one second so a lot more. The data shows that integration tests are 2.34 times slower than controller tests. That's a huge difference and running that on your app would make you sad. Now that I had proved that integration tests were in fact definitely slower than controller tests and I had nothing to do specifically with Basecamp's code, I wanted to be sure that this was still true with Rails Master and Ruby, the latest Ruby which at this time was Ruby 220. Improvements to speed are being made all the time so it's a really good idea to be sure that your problems still exist on the latest versions because you don't really want to redo work that other people have already done. I ran the benchmark script again with Ruby 220 and Rails Master and they were about the same, a little bit slower but nothing really that we need to be concerned about. If you check the iterations, they're basically the same and have a slightly higher percent of standard deviation. With Ruby 220 and Rails Master, integration tests were 2.53 times slower. Here the integration tests are in pink and controller tests are in black. When we compare the difference between Rails 4.2 and Rails Master, the two versions were really close in the number of iterations but the newest Ruby and Rails was a little bit slower. For integration tests, we're getting about 395 iterations in one second and getting 998 iterations of controller tests on Rails Master. This demonstrates really how much slower integration tests are than controller tests. We weren't seeing this when we were using time as a baseline. So now we have a solid baseline and we can start improving our code. Having this benchmark script means that we can be confident that we improve the speed of our code and prove that to others as well. The next step is to find out what's slowing down our code. We can't just go changing things without knowing what the culprits are. The first gem that I used to profile the integration test stack trace was Ruby Prof. Ruby Prof requires Ruby 193 or higher and it's faster than other Ruby Profilers because it's a C extension. Ruby Prof has been around a lot longer than other Profilers first being released eight years ago. This is the first script I wrote to profile the integration tests and find what was slowing down Rails. To write this I followed along with the first Ruby Prof example on GitHub as one does when they first start using a new gem. First I set up the documents integration test so I could specifically profile the integration test stack trace. Here I'm using the Ruby Prof API to run the code, run the index test in a block. This will capture the integration test stack trace and then finally set the printer to the default flat printer and output to standard out. And so then you can just run the Ruby Prof script in the terminal and it returns the following output. Ruby Prof has given us a lot of information and I'm not sure about you but when I first saw this I was pretty confused. I wasn't sure what I was even looking at and I had no idea how to use this to speed up integration tests. It felt like some kind of secret code telling me where the slowest methods were. Let's break down what we're looking at. First we have the percentage of time spent in this method. This is calculated from time spent in the method divided by total time. Then we have the total time spent in this method and it's children. Next we have time spent in this particular method. Then we have the wait time spent in this method, the time spent in this method's children. The number of times this method was called. And finally the method name that was called. Unfortunately even after breaking this down I wasn't entirely sure what the data was telling me. I felt overwhelmed and frustrated. I started reading the documentation more closely and playing with available options. There are a lot of different printers that you can use with Ruby Prof so I encourage you to experiment with all of them. First I found a new type of printer called Graph HTML Printer. There are no screenshots of these or how they work so I just was doing trial and error. Instead of printing the standard out this will print to a specified file and if we open that in our browser we can see that this happens. It's a bit more readable but it's still kind of confusing. And if you just look at it without really reading it it kind of seems like we're global no method is a problem. We're spending 100% of our time there. But that's probably not the case. And a better way to demonstrate why that's not the case is using the call stack printer. This was the easiest output to understand. When it runs it creates a visual display of the stack where your code is spending the most time. And if you look closer you can see that it starts at the top with 100% global no method. And this makes it clear that the reason we see that in the other one is because this is just the first thing that's called. You're always going to spend 100% of your time in the first thing that's called. So what we want to focus on are the parts of the call stack that jut out with higher percentages like this one. It shows that we're spending 12.56% of our time in action dispatch runner reset. That's not part of that first main cascade. Like it sticks out farther which means we're spending more time there than we need to be. And this is a good place to start looking for problems in your code. So Ruby Prof is a great tool for showing you the big picture but at the same time the data can feel incredibly overwhelming and frustrating if you're not familiar with how to use it. There are a lot of options that come with Ruby Prof so it's worth trying them all out. Overall, Ruby Prof was most useful for finding the differences between the stacks for controller and integration tests. In the end we didn't actually use it to improve the speed of the code. Once I had these results I reached out to members of the Rails core team to give me some insight into the data I had collected. I got a lot of feedback and at this point Aaron Patterson chimes in and says, oh hey, speeding up integration tests sounds like a really great idea for my RailsConf talk. Let's work together on this so that I can use your code in my slides. So the first tool that Aaron introduced me to was StackProf. StackProf is written by Amon Gupta and is a Ruby profiler like Ruby Prof but it takes a sampling of the call stacks so you can focus on the biggest problems and slow downs in your code. Let's take a look at how to set this up. Here's an example of the StackProf profiler code we used. Again we add the integration test code like we did in previous examples. Then we call StackProf and set a file for it to output to you. Inside of the block we run the mini test method 3,000 times. We need to run it so many times so we can get a real sample of our Ruby code. And we can run this like we did before and since it's outputting to a file we're gonna need to open the specified file after the script runs to view StackProf output type StackProf dash dash text and the path to the dump file named in your script. This will show you the sample profile of your Ruby code. And this seems a bit confusing at first. It's similar to what we saw in Ruby Prof which is all this text. But the stack is sorted by time spent in each frame at the top so you know that the top is gonna be your biggest problem method. And if we look at each column represents the first one is the total number of samples where this frame was in the stack. Next is the percent of time all samples spent in this frame. Then StackProf shows the number of samples where this was the top most frame. Then we have the percent of time where this method was the top most frame. And lastly it shows the method that was called in this frame. Since StackProf output is sorted by most time spent at the top we can focus on the top line which says we're spending 53% of our time in many tests runnable on signal. Let's take a look at that on signal method. So basically this method is just yielding. So if we wanna make sure the integration tests really are that slow like that method is actually slowing us down that much we can just delete all the code and then just yield and that's it. So there's no way that this is going to be so there's a slowness in our code because if it was another method that was yielding to there that's the method that we would see as spending the most time there. So we know that this is the problem. And so if we run the StackProf script again we can see that many tests on signal is no longer in our stack and the new slowest code is search profile at 14.7%. So we're done. This is many test problem now. So let's open an issue and we'll grab a beer and go work on something else. Oh, but we forgot to do something. We forgot to follow the number one rule of verifying that we improve performance. I call it ABB, always be benchmarking. Every time you make a change to improve performance you have to benchmark your code to make sure you made it faster. So let's just be sure let's run that benchmark script again with the change that we made to many tests run on signal. And we should see a massive improvement. Now there are not spending 53% of our time in this in many tests. And when we run this we can see that, huh, it's still 2.53 times slower. It hasn't improved at all. So after a little bit of research and consulting the creator of StackProf we found out that there's a little bug in OSX that causes CPU time to be completely inaccurate and not correctly profile the code. So this is something to remember if you're using StackProf or any other profile that defaults to CPU time, Linux doesn't have this issue so you're free to use CPU mode with StackProf and that OS, just don't use it on OSX. So we can simply resolve this by using wall time instead. So if we change the mode in our StackProf script from CPU to wall time we get accurate results that tell us where the real problems are. And that problem was not in many tests. It's not even represented in this stack sample. StackProf and wall time actually pointed to our regression in the delegate method where we're spending 28.3% of our time. Aaron changed the delegated methods to be in route set to be written out and so if we change that and we run the StackProf script again we can see that module delegate is no longer in the list and the slowest sample we have now is only 4% of time spent in per thread registry. Now we need to follow our number one rule of always be benchmarking. So if we run the benchmark script again we can see that integration tests went for being 2.53 times slower to 1.4 times slower. That's quite an improvement. We're doing a lot better now. Here we can see that we lost some iterations in the controller tests but the integration tests have gained about 200 more iterations per second. We can see that we're definitely improving the speed of integration tests and slowly closing the gap between those two. StackProf is really great in helping point out slowdowns in your code quickly because it sorts by the slowest profile rather than giving you all of the information at once. But an issue that we had with StackProf was that it was hard to pinpoint problems when methods were dynamically created or modules were anonymous. In fact, we wouldn't even have known there was a problem with an anonymous module if it weren't for Ruby VM's stat method. Ruby VM stat is a method in the Ruby Virtual Machine that provides information about the current method cache. This is useful for being able to see if your method cache is being busted. Method cache is important to the performance of Ruby code because creating a new global as a constant is very expensive. Here's an example of the script that we used to track the global and constant state of objects in our tests. First, we add the same integration tests for the index method like we've done with previous examples. Then we output the Ruby VM stats before we run the test, run the test again and output the Ruby VM stats after the test runs. And if we run the test with the Ruby VM code we get the following output. What we wanna focus on is the global constant state which tells us that before the test ran we had 7,823 constants but after we had 7,824. And this means that we're creating a new constant method while running the single integration test. So if we ran this test five more times we'd have five more new constants. Creating constants while running the test is expensive because each time when we bust that cache Ruby has to resolve the constant again every single time. We found that the method that was creating an extra constant was URL helpers. Now finding that method took a lot of digging. We couldn't rely on stock prof or standard debugging tools to find it because the module is anonymous. We had to go spelunking and the Rails internal is to find it. And there it is, module.du do. And I looked back in the githist tree and found this method used to be memoized. So I was like, just add the cache back. Why would you remove it? The diff is much larger than this but you get the idea. URL helpers is now memoized and we should see speed improvements in our benchmarks. After caching the URL helpers we ran the benchmark script again and saw a minor improvement integration test from being 1.4 times slower to 1.36 times slower and we saw an improvement in the controller test as well. But Aaron pointed out there's a better way to fix this besides caching URL helpers only like that. When working with performance it's really easy to think that you should just cache all the things because caching's just gonna fix all your problems. But caching isn't a free solution to all your performance problems. It comes with a cost and if you're able to speed up your code without caching you should do that first and then cache later. You don't want you putting a bandaid on problems that you have in your code that you could fix easily. Optimize first, cache second. The problem was that URL helpers was being called when it wasn't necessary. So integration test didn't always need URL helpers when it was being called and initialized. So the better way to fix the problem is to move it where it's needed rather than caching it. So this diff is large but if we look closer we can see that removing the URL helpers include from when the integration session is initialized to only be called when it's reset. Now that we've done that let's run our Ruby VM script again to check the global method cache and we can see the global constant state remains the same at 7,822 before and after the test runs. So if we run our back to our script again we can see that this improved integration test even more from when we cached them. When we cached URL helpers we were at 1.36 times slower but after this change we were at 1.12 times slower than controller tests. This proves how expensive busting a global constant method can be. This is a massive improvement for a fix it seems kind of minor. Ruby Proff, Stack Proff and Ruby VM were instrumental in helping us find the largest problems in Rails source that were causing slowdowns and integration tests. Of course we made a lot more improvements that I didn't demonstrate today but we were starting to get to the point where we were making micro improvements to performance and we weren't seeing those big jumps from like 2.53 to 1.4. One thing we noticed after removing the largest culprits was that we were spending a lot of time in garbage collection 11.13%. This is because we were creating a lot of objects and you need to be garbage collected and to figure out why we turned to allocation tracer. Allocation tracer was written by Koichi and is a tool for finding how many objects are being allocated by Ruby and where they are in your code. Allocation tracer focuses on a specific problem in your Ruby code and helps narrow down unnecessary allocations of objects. Here's an example of the allocation tracer script we used to track allocations in the integration tests. To use allocation tracer we need to include object space then we define the integration test as we've done in the past scripts. We then set up allocation tracer and call path line and type. This allows us to define the information that we want from allocation tracer's output. Then we assign the tracer result. Again we run the integration, we run the test 3,000 times to get an accurate sample of allocations and finally we resort the result by the number of allocations. Reverse puts the highest counts to the top and we return only the first five because we don't need all the allocated objects just the top five highest offenders. And then we run the allocation tracer script like we've done with the past scripts and the output is a little bit confusing at first but once you get the hang of the data you're looking at it makes a lot of sense. This shows us the path to the file that allocated the objects, the line number where the objects were allocated. Next is the type of object that was allocated. This can be a string, array, struct, hash, et cetera. The next array gives us the total count of objects created, the old count of objects which are objects that haven't been garbage collected yet, the total age of the objects, the minima age of the objects, the maximum age of the objects and the total memory size. Specifically we found that a lot of the object allocation problems were in rack and not rails. At least the top five biggest offenders were when we first ran this. So one of the ways in which we reduced time spent in garbage collection in rack was to reduce the number of string objects being created. A simple way to do this is to freeze certain strings. After each one of these improvements we check our allocations again and here we don't need to be benchmarking each change because we're not specifically trying to improve the speed of our code but rather reduce the time spent in garbage collection which should reduce the time our code takes to run because then we're not wasting our time in garbage collection. In this run we can see that rails has replaced three of the rack allocations in our top five. This was from changes where we first used the strings and the reason that we froze the strings was because when you assign a string key to a hash the string is copied which creates more objects. Freezing prevents this but it also makes the string immutable. Now before you declare that winter is coming and that you need to freeze all the strings you must first prove that string allocations are a problem in your code. You don't want to prematurely optimize this code because you're making the strings immutable. This isn't necessary if it's not slowing down your code and not good if the string needs to change. This should only be done if absolutely necessary so I hope I don't see any of you send a giant pull request to rails with all the strings frozen telling them I said to do it. You have to first prove that the string is a bottleneck before freezing it. Allocation Tracer was a great tool for finding where objects were being unnecessarily allocated in rails and where GC was wasting our time. We unfortunately didn't see much improvement from this but it's also because these were micro improvements in performance which is tedious. Comparing integration tests to before we started to where we ended up were almost faster than the original controller tests. Originally we could run 393 iterations of integration tests now we can run 828 iterations of integration tests. It's a huge difference and calculates the integration tests being almost three times faster in current rails than they were in rails 4.2. In fact a lot of these changes didn't alter behavior and were back ported so you actually can get them in the newest rails 4.2 and you don't have to wait for rails 5 to take advantage of these optimizations. I hope this helped you better understand your tools and how to improve performance. Be sure to know the advantages and disadvantages of the tools that you're using. You need to understand the limitations so you don't end up frustrated with your tools. No tool can do everything so be sure to use multiple tools. We were only able to speed up integration tests by combining benchmarking P's Ruby Prof, Stack Prof, and the Ruby VM and allocation tracer. And of course, always beat benchmarking. It will help you be confident that you're actually speeding up your code and able to defend it, the changes that you're making. It may even help you find a bug in your profiler. I want to say thank you to a few people who helped me with this work. First, thank you to Aaron Patterson for working with me on speeding up integration tests. I know there was some concern that he was serious when he said he stole his code at his RailsConf talk but it was truly a joint effort. And thank you to all the creators of Gems we used and thank you to FullStack Fest for having me speak here today and thanks to all of you for listening.


Hi, hi everybody, thank you for coming to my talk. My name is Dmitry and I create software that works in some cases, or last few years I'm working in the field of API testing. I built tools that aim to make it a simple and effective process. For example, I implemented one of the most performant Jesus schema editors in Rust and created schematics, a project that automates API testing to a single command. Also I contributed to popular packages like hypothesis and Django Mani. Today we are going to talk about effective API testing and let's start from the opposite. What happens if it's ineffective? First of all, logical errors. For example, if you had bad input requirements, then for example, the user can top up for a negative amount of money. Then all data API docs can be a huge source of confusion. Teams working on the project are out of sync and it becomes harder to grasp what's going on. Security issues are also possible. Last year's research from Microsoft shows that at least four security properties should be in place, but of course there are much more of them. For example, if your input validation doesn't work well, it's possible to create some payload that will slow your application down and eventually it may lead to denial of service. And of course unexpected inputs might lead to crashes. Besides all of these issues, we have a lot of time wasted, higher cognitive load and potentially even financial losses. But how did we get here? Modern systems are complex and to test them, well we need to balance resources, time and system complexity. And as practice shows with all the trade-offs, the Murphy's law works. Anything that can go wrong will go wrong. So we somehow understand the problem and how we got there. Here are some common solutions. I've heard things like less hire more people, less overtime or less just buy more powerful hardware. And I think that is quite a superficial way to solve this year. Hiring more people increases the amount of communication. Fred Brooks formulated it a long time ago. What one programmer can do in one month, two programmers can do in two months. Then other times, like the law of diminishing returns generally holds here. The more you work, the less productive you are and even worse, it often leads to burns out. Then better hardware, maybe but it's better to know why exactly you need it and if it will work in the long run. All these practices may work short-term, but all of them will likely to have some negative consequences if applied blindly. But a good thing is we tend to formalize testing processes. It gives us our pressure, precious structure, but people are incredibly good in forgetting things and we are still limited by our imagination. Nowadays there are many tools that can automatically generate schemas or documentation from the application code or vice versa. It surely removes a lot of headaches, but often it doesn't work well. For example, the fast API framework doesn't always generate fail-to-open API schemas and so on, there are many cases of that. And there could be a more effective solution. It's better to start from the design stage. So we try to prevent some issues in the first place. We need to reduce the costs because our time is the most valuable resource and we need to automate as much as possible, which is somehow connected to costs. My opinion of most important principle in API design is to make it hard to miss use. And here are some example where we can prevent problems. Let's take a look at the, one of the most popular serialization for us, YAML. Here is a list of countries, but when we try to deserialize it, we might get not what we expected. It also applies to strings like yes, no, now and so on. So these we need to escape strings and then it's parsed correctly. And then maybe we can just use more reliable information from us like JSON. At least we will know that there was no transmission errors and we don't work with a partially written schema. We can also prevent typos. Here we have a schema that accepts an object with the success key and expectedly these value passes, but success result the last letter will also work and it's not what we might expect. So to solve this, we can add a list of required fields here or in this case, like everything will work as expected, no type of one possible addition, or alternatively we can restrict additional properties and this key will remain optional. So by balancing strictness and flexibility, you can prevent some erroneous inputs. Another source of problem is regular expressions. Do you remember the Cloudflare outage from 2019? It was caused by certain kinds of regular expressions that are very resource-intensive. For example, these regular expressions took about one and a half minute on my machine and I bet that a string of 50 letters won't be processed in any reasonable time at all. So please pay extra attention to regular expressions and if possible, use alternatives so you can express your API constraints or use regular expressions engines that grant you slender complexity. So now let's take a look at how do we test things. For example, we can check that the community property of addition works. With the example based approach, we can write something like this in Python. That's a function that accepts two arguments and revise the property and we have our test cases with PyTest. So we write them manually. Of course, it's super straightforward. But again, we can test only for things that we can think of and often many H-cases are missed. In the end, we need to maintain all these when you are in test cases which also adds some maintenance costs. There's a different approach that looks at data and generates test cases instead. The most popular library for product waste testing in Python is hypothesis. In this example, we test, we reuse a test function and add some data generation strategies from hypothesis and define our numbers as integers or flaws and connect them with the test function arguments. So the run might look like this. So running this quickly reminds us that the community property for addition works only on real numbers. That's something that we might not expect initially. And this property-based testing, you will have a high variety of input data which is in many cases can be inferred from input types. Then there is test case modification called shrinking. You will have a minimal examples instead. Also, often it expands your understanding of how your code actually works by showing you many, many corner cases. But generally, it requires some guidance. You need to define your data generation strategies unless it's kind of being automatically inferred. These two approaches work perfectly together. Use symbols and getting back to the API testing topic. Actually, there are a lot of properties we can use for a property-based testing. For example, we expect that our API won't crash. There will be no unhandled errors. Then in general case, we expect that the input should match the schema and it should be accepted and errors should be rejected in general. Of course, there are some general limitations to it. Then we expect that responses match if there's definitions and all examples are working. And so on, there could be much more of them including expected response time or authorization on certain endpoints and so on. So sometime ago, I started working on a tool that combines API schemas with property-based testing. It's called schema disease. And here's how it works on a toy example. Let's say you can create a user and need to pass a name as a payload. So something like that. Then schema disease-stake the API schema creates hypothesis strategies for all API operations defining the schema. It makes progress to the API and verifies responses. So it supports three different schemas. Supports open API two and three. And also we support GraphQL. And generally you need only a valid API schema to make it work. But actually not necessarily valid. Here's some statistic on that. There is a project called Open API Directory which is a collection of various API schemas. And at the time of writing, they had 3,225 Open API schemas which are mostly syntactically and semantically valid in terms of individual keywords. Some of these schemas are incomplete. They contain references to other files that do not exist. But schema disease can process the most of them including semantically invalid or incomplete schemas. So if the issue does not affect the data generation, we can work, no problem. And in terms of API operations, schema disease works on more than 97%. The rest goes to some cases for recursive schemas with some page cases I would say. Logically unsatisfiable ones where you just can't find a fitted example. Fitted in example. Also to non-platin regular expressions and some similar issues. But there is a room for improvement for sure. Let's see how tests look like. Here I will show you a small Python test that utilizes our PyTest integration. You can load your API schema from the network file or dictionary or any readable buffer. Then this test will exercise all endpoints with built-in checks. Schema disease has five built-in checks at the moment. They cover server crashes and API schema components but you can easily write your own. Running this example, you can run it with PyTest. It will show you, in case of error, that there is a failing operation here, post-users, and it can record multiple failures per operation. So here's an error message, response payload and a Python snippet that will help you to reproduce the failure. Soon it will be possible to have a curl command instead. Option. So these are unit tests. They are usually very fast to run. They are pretty easy to customize. You can use regular PyTest fixtures, like what is strategies here are regular Python things. They are pretty easy to integrate with the rest for the same reason. And Schema disease will also test explicit examples defined in your schema. Maybe if you use dreads, we will do the same thing but also you will fill into places in these explicit examples. So you don't have to write them completely. So here's an example of payload that test generates on the previous slide. Or if you don't want to write any Python code you can use our command line interface. This command will run unit tests for all available operations in the given schema. There is also a Docker container for your convenience and the loader for command line options to tune the behavior. And Schema disease also can generate sequences of API calls. Let's extend our example API. So we create a user and we get an ID back. Then you can get information about this user by passing this ID as a path parameter to another operation. And you get some information back. Schema disease utilizes a very cool open API feature called open API links. Here you can specify how different operations are connected. Let's say when you get a 201 response on user creation then you have a link to another operation called get user. And you need to pass the user ID parameters there. We should extract from the response body by using JSON pointer syntax. And basically you extract the ID key value. So Schema disease will automatically generate API calls like this. And we try to create a user. It will take, if it's successful, then it will take the ID and it will call connected operations. The more connections you have, the more different random sequences of API calls will be generated. Use all dependent calls in place. For example, if you have a connected update operation then the created user might be updated before the get call or vice versa. And there are like all combinations possible. Let's take a look at how we can write a test with our Python API. The implementation is based on hypothesis state machines that describe all possible transitions within the schema. Let's say you already loaded your schema as with the code from the previous example. Then you need to create a state machine first and expose a test class, which is a regular unit test test class. Then you run it with spy test. And in case of errors, you'll see a report like this. Error message, response payload, stuff like that. And additionally, there'll be a Python snippet that you can basically copy paste or produce the failure. Usually it's pretty verbose depending on your API complexity, but you should be able to run it as is. So integration tests, they are slower because they utilize features like swarm testing that is laughably effective in finding bugs. It exchanges some performance to higher found defects rate. You will need to specify connections with the open API link syntax in your schema, or you can do it programmatically in your test. There is an API for that in schema thesis. But it may uncover bugs much deeper in your application. The statistically are almost impossible to find without this approach. So why you might think about using schema thesis? If you want to pay less efforts for finding bugs, the simplest way to use it is to run a short CLI command. It may find a lot of complex problems and edge cases. For example, schema thesis knows how to maximize arbitrary metrics like response time or response size thanks to hypothesis targeted testing. And it allows you to find denial of service attack and amplification attacks. So also you'll have a great diversity of generated examples that will cover a lot of valid and invalid test cases specified in your API schemas. I will tell you about invalid cases in a bit, but generally you can easily generate custom data like CSV files and so on. There are five built-in checks as I said, they allow you to check the API schema for compliance with implementation. For example, of status codes, response schemas, required headers, content types, and so on. Here we have a pretty flexible data generation. You can use any existing hypothesis strategy and tune individual request components like body query, headers, and so on. You can specify your explicit examples or actually, schema thesis will fill the gaps. There is a command line interface, you can record your tests and rerun them later with VCR style cassettes. For Python users, you can, it's possible to test ASCII and Whiskey apps directly without hitting the network. There are a few teachers in progress at the moment are working on negative testing. It'll be possible to exercise negative testing scenarios or you can create, it will be possible to automatically generate e-smail or binary formats like images and so on. And at the moment, I'm involved in academic research about the API fuzzing, fuzzing in operation with SAC, who is a hypothesis core developer. And the main question that we want to answer is whether this is effective at all. And the short answer is yes, but actually the long version is that it depends on many factors, including schema complexity, how precisely it defines inputs, and so on. We are still running our experiments and evaluating the results. So this unit includes around 20 open source projects and eight different tools, including recently open sourced Microsoft's wrestler project. These results are in progress and cover only a subset of what we planned. This part represents only schematysus, which works in all, on all test services in a reasonable amount of time and hardware resources. So far we tested 18 services and there are around 1,000 API operations of which we crashed around 20%. These projects are pretty popular, I would say. And found various non-conformance issues on around 30%. Some of the crashes are caused by the same application called pass, as well as some non-conformance issues are caused by the same schema components. And the duplication and categorization is in progress. These issues have different severity, of course, but they all become more visible with schematysus and it's up to project maintainers on how to handle it. But anyway, I plan to report them in the future and open-source the test and the self. So here's a little glimpse of what will happen next. Maybe we will put hypothesis trust. I'd like to say that generally speaking, hypothesis handles general case all standardly well, but for complex data structures like complex API schemas, there are some limitations. There's already some work going on. We have a core engine that is written in Rust, but it's used only in the hypothesis for Ruby version. But the long-term plan is to make it usable for the Python version as well. In the end, it will benefit the whole local system. Also, I'm building schematysus as a service where you will have dynamic recommendations on approving your schema, it's much faster. We will have automatic links inference, so you don't have to use open API links. There will be testing API callbacks, repor schedules and usual things for a seamless CI integration. So to summarize, keep in mind that a good API is hard to use in the first place, that you can prevent many problems upfront by designing them right. Property-based testing is effective for finding defects in web applications. And try out schematysus, let me know what you think, and please consider supporting our work. Contributions and donations are welcome. Here are some links. You can find schematysus on GitHub. Feel free to write me an email or ping me on Twitter. Here's my handle and schematysus as a service will be available here. It's all I have. Thank you. Thank you very much for having me. Thank you very much Dimitri. That was an excellent talk on an important topic. We had one question in the Q&A, which was from Kasai, I apologize if I mispronounced that. So was this only for testing of Python apps? I know one other person put a comment in there, but I'd like to hear your view in the two minutes we have left. Definitely not. You can use it with any application because basically the schematysus is written in Python. It is partially, but it works over the network. So you can run a Docker emission schematysus and put your API location as an input and it doesn't matter in what language your API is written in. And you mentioned a container so that people that are not Python programmers can do that easy way for them to onboard and use schematysus. Yeah, that's right. Also, you can use Microsoft Raft, I believe it's called. It combines multiple different tools and it includes schematysus and they use our Docker images directly. So yeah. Okay.