Hey guys, it's Nate, aka The Foot Account, and welcome back to the channel. Today we embark on a journey which is Black Friday. But what is this promo that we've been selling players from our teams for, because we expect those player prices to drop, that we've also been buying players for, because we expect those players to rise? We've been doing a lot of preparing for this promo already, but what is this promo? Whether you're an experienced all-time team player, or this is your first year of Black Friday on foot, we're going to break it down for you today and get you in the know about why this promo has so much hype, and what you need to be watching out for, even in the next couple of days heading up to Black Friday. So if you're excited for it today, drop a thumbs up, and of course, subscribe if you're new. Before we get into that, we're going to look at yesterday's content, because it was a busy day with that big icon pack. But we also had some other SBCs that we need to cover. Yet again, another FC Pro Live player via SBCs, Jen Rowe, the Australian mad lad, 3-star, 3-star, I think they gave him a nice boost. It's a decent card. I hope he gets some upgrades. I think it's the weak foot and the skills for me, and the medium, medium work rates that just don't make it that great. But the stats are all right, and he is 84 rated, so if Mark 11, the FootWiz Pro, is able to get him some upgrades, he could actually be going to an 86 straight away, because a plus 1 inform upgrade to an 84 would take him actually up to a plus 2 86 rated, which could make him Hullet Gangs straight away. So this is an SBC that might fly under the radar a little bit. It's 33,000 coins, and it's very downvoted, just because I think the Lynx, he does link to Soco, the Road of the Knockouts player, but other than that, there's not a ton of hype for that SBC. But it's one that you may not want to sleep on, just in case there are some upgrades that end up dropping. Yesterday, we also had the Fizzen, I think is how you say his name, or Ph Zinn from IACS, the Pro Squad. He is another piece to the puzzle that we are building in objective, so make sure you get that SBC done. But the big boy SBC yesterday was, of course, the Max 89 icon upgrade pack, which I did it straight away like literally five minutes after it dropped. It's going to refresh later on this week, of course, perfect timing for the Black Friday promo. And yeah, it's a very upvoted SBC. It was 88% right away when it dropped yesterday, and it's upvoted because it's cheap. What a price for this icon SBC that I was expecting to be 200,000 coins. But of course, what do we mention a lot of times when an SBC like this is cheap? That usually means it is not worth doing. Guys, the pack weight on this pack is horrific. Like it is last week on Sunday with the Hero Pack. They actually had some really good pack pulls. We opened them on stream. They were really, really good. But this week with this pack, no. I mean, some people got some good pulls, but the consistent weight of the packs that we were seeing was absolutely terrible. And since there was so much hype for this SBC still, we saw prices move kind of like we expected. Fodder did have a rise, but it didn't last long. But icons on the market, excuse me, they did rise a ton. I'm looking at guys like Butcher Gweno, Kaka, Butcher Gweno and those higher tier icons that were the most popular cards, the biggest cards you could pack, right? That everybody hoped that they would be packing. Those cards actually had some pretty good price rises up afterwards. After the SBC was dropped, because the pack weight was so bad and people had to go back and buy the cards that they were hoping they were going to pack. Now, Fodder is a really weird story because Fodder was on its way up and it was going to go even higher. And it did go up, right? I mean, 86s went from 11k up to 12,000 coins. Since this was only an 83 and an 86 rated squad inside of the SBC, it was 86s, 87s and 83s that had the biggest price movements yesterday. But they didn't go up for long and they're actually even lower now because EA dropped lightning rounds. You can see 83s, 1.4k to 1.8. Who would have thought that 83s would have been a great investment for an icon pack? Probably not too many people. But that part of the fodder market that impacts golds, guys, Fodder is really low. Look at 88s, 22k, 89s were 33,000 coins. They almost still are. 90s are below 50k. People are still buying up this fodder as fast as they can because prices just seem to drop ever so slightly every time EA run these lightning rounds. The good news is, if you have a lot of your fodder, like I still do, I haven't sold anything, guys. I still got my 88s that I bought at 22,000 coins on bid when they were 24k. I thought I was getting a great deal. Now I'm just getting at the same market price as what they actually are. I bought 89s at 34, that's exactly where they are now. We're holding, right? Because we've known all along that our investment in this fodder is more so towards the end of this coming week with all the SBCs coming for Black Friday. And what the lightning rounds that EA have put out have actually done is, it's created a really good opportunity. Like, I'm going to go in here and start stocking my club with some more 85s because I need some more of these 85 ratings. And right now, according to Footbin, they are 6.5k. 6.5k for 85s is mental. I'm going to go in club stock on the lower tier. I'm looking at the 85s, some 84s as well. If you get 84s below 3k, that's the part of the market. I think you can still buy it right now. I mean, you can still buy all fodder right now. It is technically low. With a play of the month in Bap, SBC in coming and with Black Friday in coming as well, fodder looks like a goodbye still. But I just wanted to talk about it today in length because a lot of us had fodder already. Seeing it go down is never good. But really, we're waiting for the end of the week to see those fodder prices rise. So it's kind of, it's not that scary when you know that what is coming is a Black Friday promo that literally every year makes fodder prices rise. Now, fodder prices that did go up yesterday were in forms. In forms were selling over 30,000 coins once again. And they would have went higher if EA would not have run the lightning rounds as well. A lot of lightning rounds this past weekend. And it's only going to get even crazier towards the end of this week. Also yesterday in all of the madness, the FC pro player live cards dropped a ton. And I actually went out to the market and bought out a bunch of these cards.  I bought four of them. I say a bunch, right? I didn't go all in. I'm now wishing that I would have. I bought two Delhi Allys, one to try out on my team and also one for an investment at 90,000 coins. He's up to 130K. The drops yesterday on these cards were insane. He went from 120 down to 90K and he's back to 125. Absolutely mental. I bought a Kolo Moani as well for 692.  So, you know, we're kind of watching these FC live cards. I would not be surprised if they would go up today after a week in the rewards since they are live items, even with Black Friday on the horizon. These are live cards that people want to buy because they have the potential to upgrade. And I'm very, very curious to see how they move today. But I think they will go up. So it could be a buying opportunity if you want to get your hands on any of these cards because they could go higher. But also I could see them going up for a little bit and then coming down as we get closer to Black Friday and into this next weekend. So there might be a couple buy windows on these, but I just wanted to talk about that and cover that because I did buy a good amount of those yesterday. Now, also in yesterday's content, we had ourselves an evolution. And it's another evolution that just seems okay. It's the Pacy Winger 2. And it, again, is only nine games to get an upgrade on the card. It's all about pace, right? Rapid and quick step as the play styles, not play style plus, just regular play styles. A pretty big pace boost on the card. But again, it's a max overall of 81 rated and a max 82 pace that you can put in this. Guys, I got to say this evolution is, we're going to do a video on the second channel about evolutions because it seems like we're going the wrong direction with these. They're still good. And they still provide so many opportunities to upgrade players for your club. But especially in terms of like meta cards, which is not 100% what these are made to be for. Let's be honest. This is more meant for fun and grindability aspect. But in terms of the power curve, it feels like some of these are starting to fall behind, especially with the upgrades that they're giving. Guys, I will say Trollsard is one of the best meta players you can put in this EVO to upgrade. And he's like 700 coins on the market. 89 pace, 83 shooting, 82 passing, 88 dribbling, 4-4 high medium, striker, left wing, not a bad card. But if that's one of the better ones that you can get from this without having to go through and jump through multiple different evolutions. I mean, this one was kind of cool. Relentless winger two and pacey winger. You could do both of those that are available right now and then be getting a card with multiple different play styles. Like that's cool. But honestly, guys, for a lot of these evos, I'm waiting. I'm sitting back and I'm saying, all right, I'm going to wait and see what other evos come out. And maybe I can do some more combinations with evos during the 20-day period, I think it is, that these evos are both live. So that's what I would kind of encourage you to do as well. Maybe sit tight and wait on those before doing too much. We also had a cup mode that was dropped yesterday. But honestly, I don't think I've ever carried about a cup mode less than this, no offense. I don't need the XP. Of course, the new season XP is coming anyway, this Thursday with the new season coming. And the packs in here are decent, but I'd rather just play rivals and play the normal other game modes, maybe even just do squad battles for evos because you at least get progress in that. And right here, you're in a friendly mode, so you don't get progress that. But I think that cup was just kind of, it is what it is. If you want to grind it, go for it. The packs are decent. If you're trying to stock up on some fodder before Black Friday, that is a gameplay grind that you could or taken. Now, let's get to the content at hand, the conversation at hand. It is Black Friday. And I'm going back to a couple of years ago to really start to talk about what this promo is about and why it's so big, right? Obviously Black Friday is big in real life because of the sales, the shopping, right? And the deals. And that's what EA tries to play on with Black Friday inside of Ultimate Team. They say Black Friday is back. One of the most exciting campaigns in Ultimate Team with non-stop content released over one weekend, which is very accurate. Like guys, this is one of the first promos of the year. It's probably like, I would say it's the first big marquee promo that everybody looks forward to. And it's just one of those promos that is massive, right? You think about like team of the year, team of the season. Like Black Friday is on that sort of level because of how many SBCs, how many packs. Like, you think that SBC section is already busy? Just wait. I mean, a lot of these are expiring to be fair. EA knows they had to get a lot of these SBCs out of here before Black Friday because it's going to get even crazier inside of this game with SBCs, content in general, and packs for sure. But that's what Black Friday is all about, right? It's about packs and packs in the store, packs via SBCs, and just everything going crazy. Of course, EA put stuff in packs as well that makes it worth wanting to open packs like best of team of the weeks and big time promos. And we have a big time promo coming this year as well. But let's start from the beginning. Let's go in chronological order, right? When does Black Friday even start? Well, in the past couple of years, it's actually started the day before Friday. It started on Thursday. Thanksgiving in the U.S. The day before Black Friday, they usually drop a pre-Black Friday promo and they release a best of team of the week. So this was the best of team of the week that they released. Last year, Neymar informed De Bruyne, Holland, Tamori, Fede, Valverde, Usman, Dembele, and this starts the whole madness, right? They drop pre-Black Friday flash SBCs as well. That's a big thing to try to get the hype going. But what you see a lot of times of these best of team of the week cards is if we know a best of team of the week is coming, they get panic sold and their prices drop a ton. Last year, I don't think there was a lot of news for Tamori. He went from 313,000 coins. He gets put back into packs. He went all in on a 223. I remember the best of team of the week last year dropping a ton in price on, like, Thursday and Friday. But then, as you can see here, actually rebounding a decent amount. They were really, really good to trade with. But this year, with the way things are looking, with the pack weight, I mean, and how supplied cards are on the market, I don't know if I would mess with that stuff too much. It's really going to be a decision that we'll have to see kind of in the moment. But think about cards they could put in the best of team of the week this year. We have an Mbappe Inforum. We've got a Holland Inforum. Fede Valverde. Virgil van Dijk. Sun Salah. There are plenty of different Inforum cards that could get re-released into packs this week. So, like, VVD, how much is he? 590K? That's 585? That is up a lot for a VVD card that could potentially be re-released into packs later on this week. If you have any of these tradeable Inforums in your club, I would be very careful, because they are going to get, some of them, are going to get re-released most likely this week and drop a lot in price. So, that's one of the biggest things that starts out early is the best of team of the week. Now, I also mentioned it already, but Flash SBCs, they'll probably start on Thursday, right? Because Friday, we're going to have a ton of Flash SBCs that you can see here. EA tweeted out the first Flash SBC of Black Friday is here. And it actually, they start Black Friday, the promo, really early in the day on Friday as well with the Flash SBC, and sometimes even with cards and packs, like way early before content, 6 p.m. normal drop time. But back to Thursday, we'll probably have some of these Flash SBCs as well, which, if you don't know what a Flash SBC is, basically think of it as one of those pack SBCs, where you go into the upgrade or the challenges section, you think about like UEFA Marquis matchups where you get a decent pack for doing a couple SBCs, or basically the same thing as these packs that we're getting for the pro players, that is what the Flash SBCs are most of the time. But, I mean, there were Flash SBCs last year on Black Friday that were completely different. Like, they dropped Flash SBCs for players, like there was Silver Stars, there were regular promo cards that were a part of it as well. You can see here, like Toko, a combi with an SBC, pre-Black Friday Flash SBCs, there was only a couple of them, right? So on Thursday last year, we had a pre-Black Friday Flash one, two, best of Team Lucas and Pax, and then I'm guessing that means out of positions, plus one, I'm not entirely sure what that is, and then it really kicked off on Friday. You have an 80-plus upgrade, an 85-plus double upgrade, there was a Van Persie SBC, 84-plus upgrade, we had that messy flashback, we'll get to the SBCs a little bit more on Black Friday the second, but it all starts Thursday with the first Flash SBCs, and the Flash SBCs are just basically packs. One thing I will tell you is get your club, another way you can stock your club are with gold commons from the top popular leagues with good links, right? I know that gold commons sell for a little bit right now anyway because of the SBCs that are available, like the crafting, how much are gold commons? They're still 500, I'm not even searching golds. Let's try that again, golds for 500, they're still into selling for 500 because of the crafting objective. So it makes some of these cards a little more expensive to stock in your club, but keep a close eye on those non-rares and rare golds from the top five leagues in the top nine nations because some of them will absolutely explode in price. Sometimes during those like Flash SBC rounds, there's players that go from like 500 coins to like 8,000, 9,000 coins as people are all sprinting to do an SBC. So just be careful, you can get banned for that stuff this year, but there could be good opportunity to stock some cards in your club and make some coins off of that, it just creates insane trading opportunities with solution trading as well. Now also as a part you can kind of see, I mean it did include some packs in here as well, lightning rounds. Guys, there's going to be packs. By the way, shout out to FIFA U team for this website, a lot of data from last year and ultimate team Black Friday content from years before, but a part of the whole Black Friday promo is the packs, right? Remember last year? This was during the World Cup, right? But the Black Friday 100 pack was something that dropped a couple hours after content on Black Friday the actual day. It was a 250,000 coin pack with 100 rare gold players inside of it all untradable. This pack at that time was something that we had never seen before. It's crazy to say this. We've already had a pack in this game this year of FC24 with 100 players in it. I'm pretty sure there's already been one. Now it's not the same as this one, but you know there's going to be, you think the store is crazy right now with all these packs inside of here and all the untradable packs. There's going to be lightning rounds every single hour. There's going to be, there might be a 500K pack to start off this Black Friday this year. It could be that crazy of a pack because it's all about EA selling packs. There's probably going to be, as you can already see here with this one, a Season Open or Ultimate Players pack plus gets you an extra pack for opening this pack already. That sort of stuff, there's going to be a lot of, not quite two for one, but a bonus pack inside of a pack. That little plus symbol on the pack design there. That's going to happen a lot during this weekend promo as well. There's going to be tons of packs, tons of lightning rounds, tons of untradable packs as well because EA are trying to get that bag. And of course, they're going to be getting that bag because not only is there going to be a best of team in the weekend packs, there's going to be a promo in packs as well. Now these cards are probably going to come at a packs Friday at the normal content drop time. That's how it worked last year. I mean, we were in the middle of the World Cup promo last year, but we had the Road to the World Cup cards. If you remember, Wynaldum, Jesus Corona, Sariamane, Hyunmin Sun, we had an Mbappe card, right? So, you know, we're going to have a promo in packs this year called Thunderstruck. This is the regular card design, but like we had last year, we had the World Cup icons in packs last year. Remember, like we had Puyol, that wasn't a really good one, but we had, I think we had a Figo, we had a Butra Gweno. It was in packs. We're going to have icons this year as well with the Thunderstruck icon cards. So promo icons coming out once again and regular promo cards. It's going to be, they always go out. They always go all out for promos like this.  Marquinhos, Modrich, Sariamane, like we named off some of those. You guys remember this team from last year? It was Hype and EA go out and try to sell the packs. Oh yeah, we had an R9, as a Don, a Croy, for Eusebio, Figo. How can I forget those World Cup icons? Oh yeah, because they didn't even get close to packing them. But that's what EA is going to do. They're going to put a bunch of packs in the store. There's going to be maybe not discounted packs, but it's going to seem like a deal and they're just going to be absolutely insane packs to try to get people to open and spend coins on it. And also, with the crazy cards in packs, it's going to work, right? It's going to work. It does every single year. If you spend FC points, there'll be plenty of opportunities to do that.  And also, with all those packs are going to be opened, what do EA do? They know that people are going to have cards in their clubs, tradable, untradable, and everything else. It's going to have to go into SBCs, right? Absolutely. There's going to be so many SBCs, and this is the whole reason why we've been investing in this fodder for this amount of time. Speaking of SBCs, there's a lot of player SBCs that are expiring here in the next few days. Ledley King, Negrado, Iago Aspas, Gordon, like a lot of these player SBCs are going away. That's to make room for an even crazier amount, more of player SBCs, upgrade pack SBCs that are going to be coming out in this next week that is going to be taking all of our fodder. Like, we go back on the FIFA U team website and look through some of these SBCs that were in here. You know, you've got an 87-plus base room in ICON. You're going to have ICON and hero upgrade packs out at the same time in this next week. You're going to have multiple upgrade pack SBCs, all the flash SBCs, right? Players, player picks, probably like draft tokens, and one thing that they always do, remember this card from last year? Flashback Messi. It seems like they always drop a massive flashback SBC on the promo Friday of Black Friday as well. Or at least some time. Remember, a couple years ago we had the Ronaldo, always jokingly called CR 3.5. It was a UVA flashback Ronaldo that was like 86 rated or something like that. We had the Messi last year. We had Benzema a couple of years before that. There's always a banger of a flashback player SBC as well, along with ICON SBCs, the promo cards from the Thunderstruck promo, maybe a Thunderstruck ICON player SBC. There's going to be like everything out. So, it's the content that makes the fodder rise. It's the reason why we've been investing in all this fodder and why, you know, we've got our club stock full, our transfer list stock full, and there's still time to get in as well. As you notice, the fodder prices are still low enough to invest in 100%. And we're going to be talking a lot more about the market impact this promo and what it means in the upcoming videos later this week. It's too much to cover in today's video, but stay tuned because we're going to talk a lot more right now, the market's still really just chilling. Yeah, it's kind of low overall on a lot of these cards, but we haven't seen a ton of crashes over the weekend on cards that, you know, I was expecting to see some crashing on. I think it's going to take some more news about Black Friday, loading screens, hype about the Thunderstruck promo. Mendy's still 1.05, like crazy, right? That's a card that needs to drop off and some panic. He might not because it's Mendy and his gold card stays expensive forever, but you know, there's still more panic to come 100%. That's our card today on Monday. That's the next question that we have to try to answer. Mendy's bring us our content tweets, right? Like EA tweets every single Monday of what content's going to be coming out today. What we're looking for is a league one player of the month card design for an SBC on that Twitter X post today, right? We're looking for a play the month in Bopay news guys. This SBC could be tomorrow, Tuesday, and that would be the perfect storm in sports because yeah, that would mean they can drop in in Bopay SBC and make everybody feel like if they start it or even if they don't start it, it's an SBC they could grind towards and work towards during the whole weekend of Black Friday and then EA goes to sell packs and there's just opportunity there. You pick up what I'm putting down. That's exactly what it seems like is going to happen this weekend. So watch out for an Mopay player of the month as a part of that tweet or probably just a part of that. I'm wondering how much EA is going to tell us about content coming this week before Black Friday because I don't think they're going to spill the beans too much about what's going to be coming, but maybe they'll talk about a flashback player SBC maybe they'll talk about icon player SBCs as well and more upgrade packs and I don't know maybe end of an era as those could be included in Black Friday as well. I do 100% know that we're going to get more player SBCs guys because we did not get a mini release yesterday and we are still short of the FC Pro Live players Vanden Boehmann and Hein the Swedish center back I think both of those are going to drop as SBCs today and tomorrow kind of spaced out one day each and probably dropped along the way with some more of the challenge SBCs to get the puzzle completed for that puzzle objective for the FC Pro Live. So I'm definitely looking out for a couple player SBCs coming in the next two days. Upgrade packs today is kind of a question mark for me because the more of to finish the whole thing off we still have a week to do this which is kind of nice I'm glad that EA put this out for so long because like yeah I need stuff I need to finish these off. We might have another upgrade pack today though I mean think about it EA is going to do everything they can this week to try to drain our clubs of fodder before we get to Black Friday so that also we feel like we have to buy packs and load up for Black Friday and I'm only going to finish not even safe packs but every pack that I open is going to be kind of replenishing the club because you're going to want to go into Black Friday with a full club that makes doing SBCs and making coins a whole lot easier so the only reason I'm doing this right now is because I'm trying to finish off the objective before we get to Friday but definitely watch out for the upgrade packs remember there was a campaign bag or campaign mixed party bag type thing that was added to the code as well kind of like the RTTK 84 plus well that SBC might come this week before Black Friday but it's definitely going to be one that might refresh multiple times per day and you know maybe refreshes every day who knows we'll be able to complete a lot like SBCs like that are going to be everywhere during the Black Friday promo so I'd watch out for that that's another reason to probably get invested in on the 82s the 83s they're starting to go up a little bit already as we saw in the graphs and looked at those I really like 84s is that like a super big hurry I don't think to get invested in those because these cards should stay low ish until we get to like Wednesday or to one of those kind of gamble packs the party bag type packs gets released this week but I would say it's probably a safer time now more than ever to continue to stock your club and I wanted to touch on something that I kind of already mentioned saving packs am I really going to save packs I might save a couple just to try out for the brand new promo just to see what it could bring but I mean guys I can't imagine that this Thunderstruck promo is going to be something that's super crazy packable think about it all the packs are going to be opened last year was so hard to pack a road to the World Cup card and they were actually really rare on the market it almost seemed like EA got the pack weight wrong last year on all the stuff that was in packs if I'm remembering correctly the pack weight was not good so we'll see what they do this year with the pack weight of course I'm not saving packs since it's such a big weekend there's going to be so many SBC's that are going to be out and available so kind of to make that decision for yourself but right now I am not saving any maybe as we get later in the week I will do that and as of right now as well I'm still trading I'm thinking that the market could rise today into a Monday so I'm watching the out of packs cards I'm specifically watching these FC pro live cards once again a lot of them are up Delhi Ali's 133K Karamoa still seems cheap Sima Khan still seems cheap enough to buy Talvan seems decently cheap I'm keeping an eye on a lot of these cards because I think they could go higher today for a short term flipping scenario these are the cards that I would watch today a lot of the rest of the market I would probably not mess with too much but that's what I'm thinking for the game today fodder probably rising up after content or even after the weekend rewards and then the news we get today but stay locked into the YouTube channel over the next couple of days because we're going to be talking a lot about Black Friday and how to prepare for it and also stay locked in on the second YouTube channel we're going to have multiple videos there going up about some of the icon packs we opened yesterday and also some of my thoughts on evolutions lately in this game but if you enjoyed today's video drop a thumbs up on it comment down below if you have any questions and subscribe if you're new it's been Nate for the account see you guys in video tomorrow peace out


Thank you very much for stopping by. So I'm going to be talking about generative testing. It's a little bit different from a lot of the testing that we do today. It has some interesting properties, as well as some interesting caveats and challenges. So first off, just real quick. Basically, I'm just some guy, engineer at PayGarden, giving me normal places. I work a lot on functional programming stuff. So we're actually a closure shop. I do a lot of work in OCaml as well. So I'm approaching this from a very functional programming mindset. And this talk has some pretty strong caveats. So first of all, this is mainly meant to be a thought experiment. So we use a lot of these techniques ourselves, but they're not necessarily things that you're going to be able to graft onto your existing code base right away. They take a fundamental rethink of the entire architecture in order to be able to pull off a lot of this stuff. Therefore, it's very well suited for greenfield projects, stuff where you can kind of come in and lay a very strong foundation in the beginning. I assume things like a functional stack, and I mean functional in a functional programming kind of way. So Scala maybe, Clojure, OCaml Haskell, stuff that lends itself to very strong, immutable, persistent data structures, even immutable databases, in this case. That'll come back to help us a little bit later. And then this is not a basic talk. I assume that if we're doing this kind of thing, we all understand how we're going to have a fully parallelized setup. So we're not running a million tests sequentially in an hour. There's a lot of parallel that needs to happen here. And that needs to happen at the browser level. That needs to be able to happen at the server level, even for your testing infrastructure. And then your runners and your reporters need to be able to run in parallel as well. With the Greenfield project and thought experiments, a lot of times I think there's value in saying, rather than figuring out what can we incrementally do better about our existing code base here? What if we were starting from scratch and we wanted to kind of create a perfect setup? And we'll never get a perfect setup, but we want to try something totally new. And you kind of see what you can achieve given no restraints. And then based off of what you can do there, you figure out how you can slowly start to graph that back and backport that onto existing code bases. So with that, kind of start off with what is testing for? I think you guys will all agree that testing is for exploring state transition space for invalid states. Is that right? I mean, it's a terrible definition, but actually it will help us kind of think about this a little bit later. So to give an example of what I mean, let's just imagine we have a symbol cart on a PsyTorn app. We just have a couple of steps, right? Step one is we're gonna add an item and that's gonna update our states. So now our item IDs, we have one item in there. And we can take that state and we say, all right, is that state valid? So yeah, sure. We remove an item, it's still valid. And if we add a non-existent item, that is not valid. So this is what I mean by an example of a state transition, which is invalid. We should not be able to represent this kind of thing in our code, usually. Oftentimes this takes a form of maybe we do several steps and then we run a in order to build up the states. So for example, I go to the site, I log in, I add several things to my cart and then I click on this thing. And really what we're doing is we're building up states until that final step and then we're running a test over that final transition where we say all right, whenever I click checkout, I should have this many things in my cart or whatever. So this is kind of useful because whenever we're traversing the state graph, there are times where we wanna actually narrow in on one specific transition. So testing today is largely kind of example-based where we hand write unit tests. And this includes things like page objects and whatnot. It also includes things where we use an IDE maybe to record our tests. And it's kind of an interesting idea because we think we can kind of manually numerator all of the important examples. Really is what we're saying whenever we write one of these tests, right? Whenever we write a cucumber test and we say, as a user, I do this and then I do this and I do this. And we kind of feel like we've covered a very important thing there. But actually there are millions and millions of variations on the thing that we're not really capturing. We're kind of hoping that we've got the general structure there. In fact, we're very optimistic about it because there's a combinatorial explosion between features. So if you have one feature, you can write a couple of tests and that's pretty easy. If you have two features, the interaction between them, you now have to write a lot more tests and three. The number of tests just kind of starts to explode. So for this talk, I'll just go over a little simple app. It's called Osmium. It's a book summarizing site. You guys recognize most of the ideas. So basically we can submit a book to this and we can give a summary of it and we can rate the book. That's it. We can log in, view our account, update our account. This is about as simple as you can get for a kind of real-worldish demo app. I think we can all agree this is a very, very simple app. This app does very, very little. And just to show you what it looks like, there is, it's meant to kind of emphasize how simple this is. I've kind of circled all of the different actions we can do from here. We can log in, we can sign up. We can view a book and we can list the books up at the top. On the login page, we can log in. We can log in, we can sign up. We list books. And then from the viewing books page, we have a few more actions. So like we can see already that there are a few actions, but overall still very, very simple app. A site flow from this might be, we can for example go from the listing books page to viewing a book. So if we're looking at the list of books, we can click on one and view it. If we are not logged in from any page, we should be able to sign up or log in. We saw that on the footer. If we are logged in, we should always be able to log out. We should be able to view our account. And then from the viewing our account, we should be able to update our account. Now if we are logged in, we should be able to submit a book. We should be able to rate a book. We should be able to update a book. So pretty simple still. We have just kind of a very simple state flow or site flow diagram here. But actually it kind of helps if we don't think of it as a flow diagram, but rather as a tree. For example, if we start off at the list books page, what options are available from us from there? So just like in the previous example, we can view a book, we can sign up, we can log in. But if we now traverse to the next step, we go to the login page, we can sign up. And then from the sign up page, we can log in or we can list books. We back up, we go to the login page and we successfully log in. We can now log out. We can also submit a book. We can list books. But now because we traversed this, that actually the state is a little bit different. Because now from this listing books page, we can log out. We can view our accounts, we can view a book. And if we're viewing a book, we can still log out. We can list books, we can view an art account. We can update the description, we can rate a book. And then backing up, if we go down this branch, we can view books, log in, list books. You guys kind of get the idea. So the idea is actually it's, when we think of it as like a site flow, we tend to think of it as like this kind of closed loop. And actually it ends up being this ever-expanding tree of state possibilities. And this is incredibly difficult to enumerate in tests. It actually, for each one of these branches, there's an infinite number of sub-branches gently. Like very few of these end up terminating. And so whenever we write or record tests, we're actually tracing a specific path through this tree. And we're hoping that kind of we've captured the canonical example. And that's gonna be enough to protect us later on whenever we do some refactoring, adding a feature of fixing bugs. And this actually gets worse whenever we consider concurrent access to a shared resource. So simple example, let's say that we have two tabs open in a browser. Tab A logs in. The tab B goes to the add books page. And it fills out the form, but does not yet submit it. Tab A logs out, and then tab B submits the book. Normally whenever we're looking through that tree, we would expect that through the linear flow, you wouldn't be able to do this, right? Because we've logged out, so that's no longer a thing that's accessible. But actually because of the concurrent nature of this, like maybe a user will do this, right? Users are encourageable, and like the way that they will find bugs in your site. And so now we have this question, like what's the resultant state? Did we create a book? Like if we have it submitted by ID, like the user that created this book, is it null? Did we write a test for it? Did anyone think to write a test for it? It's kind of the challenge there. So an alternative is just not to write or report tests. In fact, you should generate them. This is by John Hughes. He's a creator of QuickCheck, which is a very famous generative testing and property-based testing library for both Haskell and Erlang. So, or alternatively, as we just heard in the last talk, we should automate all the things, even writing the tests. So the core proposition that we're gonna say here is that it is a much better use, or it's a much better idea to use infrastructure, computers, rather than Cuban times, Cuban time. Computers are really good actually at enumerating all of the branches of possibilities, much better than human beings. We tend to lose interest if we say, all right, what if I click login seven times and then click log out and then click login again? Very few people write that test. But the computer doesn't mind, the computer will do it. And in fact, if we can teach the computer to do this, then if we have, if we write properties about our tests, for example, that we say, whenever the user is logged in, they should always see a log out link, right? That now generates, or we can now generalize that over, sorry, if we take that property and the ability to walk through the site and to generate all these different examples, that one property now is able to expand and scale over millions and millions of tests. So we get kind of constant effort, no matter how many tests we run, versus the linear effort that it takes, if we want to run a million tests of handwritten ones, we have to write a million tests. So this scales much, much better for covering all the state space. So there are kind of two properties of properties, two classes of properties that we can assert. One is about, no matter where we are in the graph here, we have certain properties that we want to assert about the state. So for example, as we're traversing it, we should never see a 500 error page, right? The user should never be able to do anything that causes a 500. Similarly, they should never be able to do anything that causes a JavaScript error. If the user is logged in, they should always see a log out link and vice versa. There are all these kind of very fundamental basic properties that we can assert, no matter where we are on that graph. And then, but more interestingly, we can actually target specific transitions. And we can say, for example, that if we're at the rate book page and we've just kind of come out of it, well, one thing we know is that the user must have been logged in in order to do that. That if we're keeping track of it, then the number of ratings for a book has to increase by one. The updated ads timestamp should have changed. So we can do a test on the properties, on the state, but we can also test on the page, on the rendering. So for example, the log out link must be visible. We can do things like, if we've updated a book, then we must be at the view book URL. After we've edited the description, it should send us back to viewing the book. Same thing with log out, count, sorry, submit book. Each of them has a specific set of properties that we want to ensure that no matter where we came from, whenever we are going into it and wherever we leave it, that there's the state conforms to certain properties and the page conforms to certain properties. So there is a challenge in teaching a computer to navigate our app. You can do things like, for example, so as we saw, every page has an inbound kind of edge and outbound edge. So all the other pages that link to it and all the pages that it links to. And the page model, as we saw and we've seen before, can explicitly list these. So we can actually encode in the page model and we can say that this page is a larger transition to these other pages. And then you can use this kind of a predicate. You can say that, given this state, here are the pages you can go to and here's how to generate the link to them. But this is pretty clunky and it's actually really prone to rot. I'm not a huge fan of this idea. If I have to go back and maintain something and if I forget to update it, and the tests just kind of silently don't cover something, then I am likely to get in trouble much, much later. So I want something where the computer just kind of does it on its own. So in our case, what we try to do is actually capture at render time. We say, given this input, which is some immutable piece of data, even the database is immutable, we say, all right, here's the input. What are the outputs that are available here? So we actually rendered the page to an intermediate form in closure, it's known as hiccup. And each of these, so rather than actually just hard coding a link inside of the page or whatnot, we use functions to say, all right, generate a button with this action name. And so we can then take that data and we actually extract that. We walk over the data structure and we say, all right, here are all the possible next transitions that we can go to based off of this input at this node. We do a couple of metadata tagging. And then what happens is, as we generate these nodes, we then walk, we keep them over to the side and we can walk down them later on and we just basically just recur. And then that's how we actually are kind of teach the computer to navigate through our site. And then each step along it away, each time we go to a new node, we can run the appropriate assertions. So now the computer knows how to render a starting point. So now we just have to list our starting points. It knows how to figure out where it can go from there. And it knows which test it should run at each node. So just giving you an example of what this might look like. This is an array that was generated by one of the tests. And it's gonna just say, at the start, go to the homepage, click on this action, fill in this password with this. You can see that in this test case, it really likes clicking on this element for whatever reason. And then it fills in the email. This is the kind of test that a human probably wouldn't write, right? It seems not really relevant to click on something five times. But oftentimes you'll find that there is some states that ends up changing because of submitting a blank form over and over or whatever it might be. So to give you an example of what a test might look like, we wanna know that no matter where we are on the graph, we should never show an error page. There should never be a case where the user sees some error. And so there's a bit of setup here. Can kind of ignore this. We're just getting a new browser and then we tear it down down here. What we're saying is I want you to walk the graph in steps. So starting from the homepage, I want you to render the page, get all of the steps that are possible and I want you to walk down each of them and then walk down each of all of those subsequent ones, five layers down. So there's this huge explosion of number of possibilities. Now each time you do that, I want you to run this function where you give me the current action and also the path that we're on. And I'm gonna run some assertions based off of where we are. So in this case, it's easy. I don't really care about the specific node and I just say, I wanna make sure that I don't find this string in any of these pages. And if I do, I wanna record the path. How did we get here? What was it that we did in order to trigger this error page? So quick demo. This has some assumptions. So we have a few fully immutable database. We have a reference to the transparent application where we have states passed in. We have the database is actually passed in. We can muck about with it as much as we want. And then if we pass in the same database later on, we'll always get the same output. And this is really important because as the computer is doing crazy things and traversing randomly through our application, we wanna make sure that whenever we replay that same path, we get exactly the same output. So in this case, I'm just gonna look at here all the failing paths that we have so far. This could be tough. I might have to talk quickly. All right, so we have no failing paths. So we're gonna start our server and then we're going to run our tests and we wanna walk 11 steps down the graph. So we're gonna go to the first page, get all the children, walk all those, get all their children, walk all of those, run all of our tests, and we wanna make sure that there are no errors at all. And so you can see actually we've hit some errors in this page. We expected not to see any errors and actually we did. So now we can take a look and say, what were the paths? And the paths actually corresponds to repro cases. It's literally saying, if you go through and you do this, you will see an error page. So here are all the different ones and we just wanna look at the last one, which will be the longest one probably. And you can see that if we just do this and you can see that it involves clicking several times here, we actually see an error page. And then you can take that and you can give that to a developer or you can put that in, you can save that for later, for a unit test or whatever. But now the computer has actually generated a valid test for us to make sure that we would not have written almost certainly. And then you can do kind of cool things where we have two different classes of tests here. I have some user tests where I wanna make sure that every time a user is created, that there is valid, that it always has the properties that I expect. And I wanna run, what I'm gonna do is create 250 threads up here. I'm gonna run that test where we walk 11 steps down the graph and then create 250 more threads at the same time. Walk 11 down those. So 250 times, 500 times we're generating the graph and walking 500 or 11 steps down and running assertions across that the entire time. So we start with no assertions yet. We've made no assertions and we have no failing tests. And this is sped up because we're spinning up a lot of stuff. This is Sauce Labs. There are some significant infrastructure challenges to running all of these tests. So you can see that it's actually hitting our site. And there are a lot of failures in this test. And I really like up here in just a moment. So you can see we start to have 286 assertions have been made so far. We have lots of failing test cases that we can actually use for repro later. And at some point here, we go from 133 to 559. Is this like we literally like break the scale? It's supposed to be halfway. Yeah, I just love this. Yeah, so I mean this is really cool. So now we actually see how many assertions did we make? We made actually close to 13,000 assertions on that when we have 116 cases that failed. So we can actually take this and those are repro cases from now on. All right, so there are a lot of challenges in generative testing though. So in particular, figuring out the right properties to test. So this requires a very different mindset because we're very good at finding examples of what we want to test. This is what user stories are all about, right? Where we have a specific story in our mind. And what you actually need to do is kind of lift that up a level and you need to abstract that to what are the general properties that we're actually looking for at any given time. And this is actually, it takes a lot of practice to get into that mindset and to internalize it. Another thing here is that reproducibility is critical. So it doesn't matter if you have that repro step. If every time you go through the site, a bunch of state changes in your database and the repro case is different every single time. You need to make sure that the app is very core reproducible. And flakiness is a really big deal here as we'll see a little bit later. Every time the test fails once it needs to fail again in the future until you fix the bug. Within a single run. Another thing is you get really long tests. So you saw that run pretty quickly. It was sped up I think three times but it's still pretty quick. But some of the tests can be thousands and thousands of steps long, right? We only went to 11 steps in there but we're blowing up a lot of browsers in order to do that. And a lot of times the thousand long step repro case, a lot of the steps are not relevant. A lot of times you click on the same element 10 times doesn't really change anything. And so if you give that to a developer that's oftentimes very frustrating. Because as a human being what do you do? You start to kind of do this binary space partition where you take out at one and you see does the test still fail and you take out another one and another one and you kind of try to shrink it. But it turns out that the generative testing people have thought of this there's this thing called shrinking. Which is pretty cool. I'll come to you in a second. The other big challenge here is the extreme scale. You are probably not going to be doing this on your laptop. In order to run, that one that I just showed will run all rights but in order to hit a million tests per hour you're gonna have to run probably 10 dozens or hundreds of servers in your testing infrastructure. And that's not just because of the test but actually because of shrinking. So shrinking, super cool idea. So we have this very, very long failing test. 2,000 steps. Which is great, right? We found a bug. We didn't have to do anything, we found a bug. But the bad news, super long. Diagnosing this bug may be longer than trying to go and figure out how to like repro it yourself. And I love this idea of like a user reporting this bug at some point. Just for having like a 2,000 step thing. And then trying to debug it. But as a human being what you would do is you take out some of the steps and you would rerun it and then you'd see if it still fails. And you would see like what are the relevant pieces here? And it turns out that's actually exactly what shrinking does and it just repeats it until it has a minimum repro case. So you can take this 2,000 step case and it will turn into a seven step case where it says here is the minimum like set of steps that I found from this long list that still produces the same error. And so this is why reproducibility is incredibly important, right? Because we need to know like if you have flakiness and the test fails for a different reason then shrinking is not going to be able to do any work at all. It needs to be able to actually find the same bug by running it again. And then kind of stepping back again to the infrastructure if you imagine you have that 2,000 step test that's a lot of subsequent tests that you then have to run to figure out what are the minimum number of relevant steps. So I think for tests, recording has its place but it's not in the browser. It's kind of in the database. So one of the nice things about this functional setup is because it's an immutable database and it has structural sharing all the way through we can take a snapshot at every step along the way. So as we're running this kind of crazy number of tests over the database, we keep a copy of every database that was generated and a share structure so it's very, very efficient. And then later on we can actually go and run tests over the state of the database. So rather than running super slow tests in the browser we can run very efficient tests in the database. So when we need to run browser based tests we wanna do page rendering and that kind of thing but we can do that and then whenever we wanna run specifically state-based tests, we can do that here. Yeah, so kind of the question is like why do we care about like a million assertions per hour and the title, it's pretty cool I think first of all. But really it's about if you think about how deep you wanna go in these tests, right? You want the computer to be able to test very, very deep in each one of those paths. The deeper you go, the longer the session you're representing. You also want them to be able to traverse over the breadth. So you wanna make sure that you're not just checking what happens if I log in, log out, log in, log out a million times. You also wanna try some things like what if I log in and then I submit a book and then I do this. You actually wanna get a breadth of coverage over as well. And then once you find these bugs, like the computer then needs to be able to shrink it. So there's a lot of assertions that happen in this. So infrastructure is a challenge here, but tractable in our case, yeah, we just, yeah. So generative testing has been used in lots of areas but mainly in memory kind of context. Nothing that's super slow, nothing that's blocking because of the huge demands it takes. So CircleCI is what we can use to scale out our servers in the test infrastructure. We use self-slabs to scale out on the browser side. And there's some pretty cool future areas here as well. So quick show of hands, anyone notice a problem like with the generative testing? Anyone think of one case where it might not be great, where it might miss some bugs? Anyone? Back there? Yeah, so there are cases where, so Erlang has a system called SimpleCheck, which is actually really good about this kind of thing. It's called linearizing the test cases. So if you see these kind of successive states that depend on one another, then you can then assert that there must have been some linear way of doing it or else it's a failure. So there are some kind of cases where this has been thought of but another kind of interesting one or technique that's based off of this is concoct testing. So it's a combination of concrete test and symbolic execution. And it's a terrible, terrible name. Like I always feel childish saying it but it is a really, really cool idea. So give me one second, let me change the color on this. One of the challenges or like one of the big cases that generative testing might miss would be something like this, where let's say that we're going, we have a function that takes a user and we wanna decide if they're like a super user based off of how many internet points they have. And we have three cases that if they have fewer than 42, they are not. And if they have more than 42, they are. And if they have exactly 42, then the whole app blows up. So we have to generate a very, very specific value in order to trigger that third case. Like we can run infinite number of tests, more than 42 and fewer than 42 and never hits that 42. So we can actually use something where we bring in code analysis. So code analysis is something, it's a bit like static analysis where we will take a function and we'll examine all of the code paths that are in there. And so in this case, we see that there are three code paths and that they're guarded by three conditions. And so what we do is we then back propagate that. What are those conditions and what are they reliant on? In this case, they're reliant on solely internet points. So on the input there. So then we will take the cases and we'll constrain what we can generate for the inputs to make sure that we actually exercise that branch. So repeat for fewer than 42, for more than 42 and then finally repeat it for exactly 42. And so we can constrain it and we can make sure that we now have three classes of values that we can generate and we can make sure that we'll actually execute every bit of our code without having to teach it how to execute our code at all. Unfortunately, so this is really cool. This is like cutting edge. There's just a recent, this is an academia still basically, there was a recent implementation of an Erling. However, this is really hard to translate into for example, a Selenium test.  So there's a lot of research left to be done on like how do we know whenever we hit a page what codes we're hitting and then how do we change the input for a page to make sure that we hit the right code paths. Another really cool thing is predictive testing. So the idea is just like we did the, we had that kind of declarative that simple data structure that was a repro case for us. We actually record the actual user's site or path through our site and then if an error is triggered we automatically email the developers without repro case. And then another thing we can do with it is we take the previous 10,000 sessions and this is something I did beforehand but we take 10,000 sessions or a rolling window of the previous 10,000 sessions and before we would push any new code we would actually take those 10,000 and replay them against the new code. And so then we would have this idea before we push this code is this new code going to break anything that our users are actually doing? Yeah, so just basically in summary humans are really terrible at manually enumerating all possible stage space and computers are really terrible at being creative about it. But they are very good if we teach them to find edge cases. So we just have to give them a little bit of help. And most important thing is don't write tests, generate them. So I'll just show you a quick example of predictive testing. So this is a little demo chat app where it uses the same structure. We're going in, we're changing state, we're updating things. And then we're going to pull up a debugger and you can see we have the same number of states here and we can just kind of scroll back and forth. You guys have seen this probably with things like, I don't know if the ID does this, but it's basically a time traveler, right? Where we can go back and forth. But if we have those state-based repro cases, we can actually take the user's session, we can load it up in our browser and we can actually scrub through it and see exactly what happened whenever an error is triggered. So there's a lot of possibility if we start to go into this generative testing model which requires everything to be kind of data-driven. So just real quick, almost all the ideas here, the work and whatnot is done by Sebastian. He's a coworker of mine, he's a cool guy. You should follow him on Twitter. My company is cool with me coming out here which it was along flights and they were super nice to let me do that. So I want to thank them and yeah, happy to take any questions. So irrespective of generating tests, but let me just decide, don't forget, I'll answer the first question first. So generative testing, so it's not about parallelizing tests, right? Generative testing is not about bringing test times down. In fact, it uses a massive amount of infrastructure. Like it is required that you already have this stuff there. What it is about is saying, I have described the general way that my application states works and I have some properties that I believe are true here and I want you to try to disprove me. So I'm gonna have the computer actually search through that and try to prove me wrong and then whenever it does find something, give me the minimum case that shows that. So this in no way is going, in fact, you need to have all that infrastructure because in order to run a million tests per hour, you need to have a lot of setup, yeah. So second question. Well, humans are definitely not doing a million tests per hour, right? Yeah, also, I think nobody's, so people do this but just not in Selenium as of yet. So because there are big infrastructure challenges around, it's a lot easier to run a million, so for example, a really kind of classic use case of generative testing is let's say you have an algorithm that you're implementing and you're gonna implement a very slow version of it but it's very correct, it's really easy to read, right? And you can kind of look at it and make sure that it's right. And then you're gonna implement simultaneously a very optimized version of it. And so you're going to say that for every input into the first function, the output should match the output of the second function. And so now I can actually use the first one as a test case for my second one. And this is very easy to do for in-memory stuff, right? Like, there's almost no latency. You could also use this for refactoring of a webpage, for example. You could say that I have an old version of the webpage, we're implementing it in a new language or whatever, it's a terrible idea, but let's say we were. Anywhere I go on the old page, it matched the new page. And you could say, try to prove me wrong. It just hasn't really been done in the Selenium world because there are difficulties in how do you actually get the browser to traverse your site? How do you make that efficient enough so that you can run a million tests per hour kind of regularly and it's not really a big deal? Like, I think a lot of us need to move in that direction. But right now everything is, and then the last bit is like everyone, like the status quo is to write your tests manually or to write specific example case or to record them. There's very little thought over the general properties of our application. So I think very few to no people are doing this right now. No, so it doesn't, in some ways it replaces it. So the way we always say it is, so we have, for example, the predictive testing. So the first thing is, we have everything our users have done beforehand. We run that through our new code to make sure that we're not breaking anything that they would want. We then have generative testing that explores a bunch of the state space. And then after that, if something actually makes it past that and we get an error from a user, so we'll get a nice email with the repro steps, exactly right, and say, here's how you get this user's case. In fact, you can load it up in your browser and you can scrub through it. Often times we'll take that and generate an automated unit test based off of that. So it kind of augments it. So most of the leverage is gonna be in writing the generative tests, but you can take the specific examples that have gone, like, have made it through those gauntlets and then just use those to make sure that they don't happen again. Say this again. Oh, yes, definitely. So there are often cases where, depending on what level you're working at, right, where maybe someone logged out, but actually there was a bit of state that wasn't cleaned up, and that was only apparent if you did a couple of other things to the site and then you submitted something over here. And like, you know, as humans, we always imagine when we write that logout code, it's so simple. There's no way there's a bug in this code, right? And then there's no way that this bug here would affect that thing over there. And so that's where a case where having the computer actually go through and say, by the way, I found this bug, which this is basically the mirror of having a lot of users, because if you have a lot of users, this will happen. They will do generative testing on your site, basically. Users do crazy things, where they'll log out and they'll go and do a bug and you'll have a hard time debugging it. Oh, sorry. Oh, so we use Clojure, and then we use Selenium. So the Clojure is a functional Lisp on the JVM, and then we use the Clojure WebDriver. Yeah, I don't. Yeah, so the code is up on GitHub as well. I'm happy to talk about the specific implementation. Like, it's not as relevant as the ideas, but yeah, we use Clojure on the JVM. We use Selenium WebDriver stuff. Someone stopped me if we're running out of time. I don't know. Yeah, so it took quite a long time to kind of think about what are the properties that I actually want to assert about my code? But I think just after you do it for a while, you get a good sense of it. It does mean that you need to be a decent developer. This is not a manual QA kind of thing. You can take the manual QA test and try to figure out what's the actual property that they're testing for? Because underneath each example, there's actually a generalized property that you're trying to prove. And so you kind of learn to suss out like, what is this thing is actually doing here? But really, yeah, so like I said, one of the common ways is to compare two alternative implementations where you're trying to do a new thing, and it has to match up with the old one. That's really critical, but you kind of, for whatever reason, are kind of scared to touch the old code. And so you just want to make sure that the new code matches the old code. So that's a really nice way to start with it. And you get a sense of how all the pieces work, and then you start to figure out, all right, what are my generalized properties that I now want to, that's entry level, and then you can go to the graduate level, which is I'm going to now write my own properties. So the unit level is a lot easier, I think, because it's generally gonna be in memory, you're not gonna be bringing up a browser and tearing it down and whatnot. So you have a, like one of the killers for development productivity is the turnaround time, right? So if it takes me five minutes to test something, I only get to test that a couple of times per hour, and that's really, really, like that kills my flow. If it's a unit test or integration test, you can probably keep that turn cycle down into the milliseconds, right? If we hit it into it, it runs a million tests, or a thousand tests, it doesn't matter, and gives you an answer. And I would actually encourage you to start there, but I think from maybe you're thinking about the philosophical direction, like how do we encourage developers to do this? I think it just goes to, like, drawing out that state's diagram was a big kind of mind-blowing thing for me when I started to realize, actually, I'm just scratching the surface here, and I don't want to spend all of my time, like with these examples, and then once you actually get a couple of test cases in, it's like going from no testing to testing, right? Where you say, actually, I did have a few bugs, and I'm glad I have testing.  So it's just kind of an experience where you have to gradually get someone in, get them to have a few success points, where they feel like, all right, I feel good with this, and then once they feel that sense, I think it becomes this obvious thing where you want to invest more time in it.  Yeah, so like I said, this is a thought experiment, right? So, I mean, it's not that we actually do this, but this isn't a promise of you can use this today in your existing code base. So there are ways to do it, where let's say you were to, for example, I don't know if Oracle has this, but if you had a mock in-memory database, where it provided the same API, this is kind of crazy. I don't think it would be a good idea at the Oracle level, but then you could keep a copy of every database. So we use Datomic, and Datomic actually provides this out of the box, where every transaction is actually kept in history, and we can always record, like, we can zoom back to another point, think of it like git, we can go back to a previous state, and we can fork the database and use a clean state from there. So we can always go back to any previous states, and then that helps a lot with the reproducibility, because we're pointing at that point in. You can see what the rate of, there's one, there's one there that, how many, like, two, six. So it depends, so it's a really good question, actually. So we actually give every test its own database, but its own database, in our case, is just an in-memory data structure, sure. So everyone gets their own environment to call it, their own kind of clean, pure environment, unless we are testing for concurrency, and we want to know what happens when we have multiple users testing on it. But that's a very explicit thing versus this, like, we'll just randomly throw everyone altogether on the same database. One more?  We're already driving Selenium through the page, and we're making assertions about the rendering of it, and we're trying to mimic real-world behavior via the browser, or via the mobile app, and kind of while we're there, we might as well make these assertions. And oftentimes, we do, we won't use necessarily Selenium to verify the status code. We'll just use that, like, we'll look up the request ID, and we'll say what was the status of that from the server point of view. So, you know, points the effort at the right level, I guess, right? Use Selenium for what it's good for, and then use the server and the back-end and memory data structures for what it's good for. All right, thank you very much.